Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 13:58:37.705528 398 test_util.cc:268] Using random seed: 499613990
I20240503 13:58:37.711906 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:58:37.712008 398 ts_itest-base.cc:113] --------------
I20240503 13:58:37.712046 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 13:58:37.712066 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:58:37.712081 398 ts_itest-base.cc:116] --------------
2024-05-03T13:58:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:58:37Z Disabled control of system clock
I20240503 13:58:37.720867 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:40943
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:40943
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 13:58:37.822280 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:37.822496 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:37.822535 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:37.825320 410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:37.825376 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:58:37.825395 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:37.825412 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:58:37.825431 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:58:37.828765 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:40943
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:40943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:37.829581 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:37.830716 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 13:58:37.839130 410 server_base.cc:1034] running on GCE node
W20240503 13:58:37.839896 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:37.840684 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:37.841514 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:37.842044 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:37.842900 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:37.844039 410 hybrid_clock.cc:648] HybridClock initialized: now 1714744717844004 us; error 55 us; skew 500 ppm
I20240503 13:58:37.845772 410 webserver.cc:460] Webserver started at http://127.0.99.190:46533/ using document root <none> and password file <none>
I20240503 13:58:37.846259 410 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:37.846333 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:37.846505 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:37.848088 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bf4f6633f97b4198b94f91a0e3b2c4fb"
format_stamp: "Formatted at 2024-05-03 13:58:37 on dist-test-slave-snql"
I20240503 13:58:37.848556 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bf4f6633f97b4198b94f91a0e3b2c4fb"
format_stamp: "Formatted at 2024-05-03 13:58:37 on dist-test-slave-snql"
I20240503 13:58:37.851608 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:58:37.853755 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:37.854662 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:37.854768 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bf4f6633f97b4198b94f91a0e3b2c4fb"
format_stamp: "Formatted at 2024-05-03 13:58:37 on dist-test-slave-snql"
I20240503 13:58:37.854842 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:37.886554 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:37.887226 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:37.887432 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:37.893671 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:40943
I20240503 13:58:37.893680 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:40943 every 8 connection(s)
I20240503 13:58:37.894584 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:58:37.896131 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 410
I20240503 13:58:37.896246 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:58:37.897003 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:37.902159 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb: Bootstrap starting.
I20240503 13:58:37.904191 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:37.904850 477 log.cc:824] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:37.906303 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb: No bootstrap required, opened a new log
I20240503 13:58:37.908974 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4f6633f97b4198b94f91a0e3b2c4fb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40943 } }
I20240503 13:58:37.909170 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:37.909231 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4f6633f97b4198b94f91a0e3b2c4fb, State: Initialized, Role: FOLLOWER
I20240503 13:58:37.909657 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4f6633f97b4198b94f91a0e3b2c4fb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40943 } }
I20240503 13:58:37.909777 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:58:37.909837 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:58:37.909945 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:37.910775 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4f6633f97b4198b94f91a0e3b2c4fb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40943 } }
I20240503 13:58:37.911087 477 leader_election.cc:302] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf4f6633f97b4198b94f91a0e3b2c4fb; no voters:
I20240503 13:58:37.911340 477 leader_election.cc:288] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:58:37.911696 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:37.911882 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [term 1 LEADER]: Becoming Leader. State: Replica: bf4f6633f97b4198b94f91a0e3b2c4fb, State: Running, Role: LEADER
I20240503 13:58:37.912127 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4f6633f97b4198b94f91a0e3b2c4fb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40943 } }
I20240503 13:58:37.912407 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:58:37.913766 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4f6633f97b4198b94f91a0e3b2c4fb. Latest consensus state: current_term: 1 leader_uuid: "bf4f6633f97b4198b94f91a0e3b2c4fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4f6633f97b4198b94f91a0e3b2c4fb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40943 } } }
I20240503 13:58:37.913901 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:37.913766 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf4f6633f97b4198b94f91a0e3b2c4fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4f6633f97b4198b94f91a0e3b2c4fb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40943 } } }
I20240503 13:58:37.914134 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb [sys.catalog]: This master's current role is: LEADER
I20240503 13:58:37.914760 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:58:37.917275 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:58:37.921882 492 catalog_manager.cc:1347] Generated new cluster ID: 5ae1a4b69be44ab88066ce13ea697dec
I20240503 13:58:37.921936 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:58:37.935600 492 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:58:37.936303 492 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:58:37.942414 492 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bf4f6633f97b4198b94f91a0e3b2c4fb: Generated new TSK 0
I20240503 13:58:37.943106 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:58:37.952251 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40943
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 13:58:38.054467 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:38.054668 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:38.054723 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:38.057633 501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:38.057693 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:38.057776 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:58:38.061137 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:38.061930 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:38.062978 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:38.071100 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:38.071586 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:38.072453 501 server_base.cc:1034] running on GCE node
W20240503 13:58:38.073480 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:38.073825 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:38.074355 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:38.075505 501 hybrid_clock.cc:648] HybridClock initialized: now 1714744718075493 us; error 34 us; skew 500 ppm
I20240503 13:58:38.077247 501 webserver.cc:460] Webserver started at http://127.0.99.129:37125/ using document root <none> and password file <none>
I20240503 13:58:38.077761 501 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:38.077821 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:38.077987 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:38.083956 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "769439c9b5d3418eb0ebd0287954e6cc"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.084401 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "769439c9b5d3418eb0ebd0287954e6cc"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.087517 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 13:58:38.089385 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.090229 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:38.090320 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "769439c9b5d3418eb0ebd0287954e6cc"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.090394 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:38.107043 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:38.107553 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:38.107694 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:38.108146 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:38.108893 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:38.108943 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.108992 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:38.109015 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.116806 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33075
I20240503 13:58:38.116828 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33075 every 8 connection(s)
I20240503 13:58:38.117655 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:58:38.123360 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 501
I20240503 13:58:38.123484 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:58:38.125586 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:38.125655 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40943
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 13:58:38.125797 629 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:38.126214 629 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:38.127640 441 ts_manager.cc:194] Registered new tserver with Master: 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:58:38.128719 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:47751
W20240503 13:58:38.227633 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:38.227826 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:38.227888 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:38.230664 632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:38.230726 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:38.230813 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:58:38.234184 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:38.234913 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:38.235935 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:38.243921 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:38.244942 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:38.245795 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:38.246619 632 server_base.cc:1034] running on GCE node
I20240503 13:58:38.246941 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:38.247431 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:38.248569 632 hybrid_clock.cc:648] HybridClock initialized: now 1714744718248551 us; error 35 us; skew 500 ppm
I20240503 13:58:38.253686 632 webserver.cc:460] Webserver started at http://127.0.99.130:42649/ using document root <none> and password file <none>
I20240503 13:58:38.254199 632 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:38.254277 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:38.254464 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:38.256014 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.256435 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.259593 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240503 13:58:38.261488 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.262391 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:38.262490 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.262566 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:38.278645 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:38.279181 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:38.279350 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:38.279851 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:38.280678 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:38.280725 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.280784 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:38.280809 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.289640 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:42209
I20240503 13:58:38.289670 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:42209 every 8 connection(s)
I20240503 13:58:38.290515 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:58:38.291458 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 632
I20240503 13:58:38.291568 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:58:38.293862 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40943
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 13:58:38.300660 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:38.300873 760 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:38.301618 760 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:38.302754 441 ts_manager.cc:194] Registered new tserver with Master: 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:58:38.303612 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:46117
W20240503 13:58:38.403380 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:38.403582 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:38.403625 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:38.406405 763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:38.406456 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:38.406522 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:58:38.409852 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:38.410588 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:38.411633 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:38.420068 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:38.420843 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:38.421599 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:38.422771 763 server_base.cc:1034] running on GCE node
I20240503 13:58:38.423117 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:38.423635 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:38.424782 763 hybrid_clock.cc:648] HybridClock initialized: now 1714744718424764 us; error 37 us; skew 500 ppm
I20240503 13:58:38.427098 763 webserver.cc:460] Webserver started at http://127.0.99.131:40559/ using document root <none> and password file <none>
I20240503 13:58:38.427665 763 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:38.427726 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:38.427893 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:38.429545 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f58fb372531d42a3bc405510bf5bced4"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.430011 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f58fb372531d42a3bc405510bf5bced4"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.433219 763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 13:58:38.435181 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.436218 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:38.436311 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f58fb372531d42a3bc405510bf5bced4"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.436385 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:38.448817 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:38.449324 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:38.449484 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:38.449959 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:38.450760 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:38.450806 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.450863 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:38.450889 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.460345 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:44047
I20240503 13:58:38.460371 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:44047 every 8 connection(s)
I20240503 13:58:38.461259 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:58:38.469413 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:38.469625 891 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:38.470074 891 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:38.470741 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 763
I20240503 13:58:38.470855 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:58:38.471189 441 ts_manager.cc:194] Registered new tserver with Master: f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:38.471812 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:47929
I20240503 13:58:38.473062 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40943
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 13:58:38.572471 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:38.572674 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:38.572747 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:38.575556 894 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:38.575614 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:38.575702 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:58:38.578954 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:38.579696 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:38.580649 894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:38.589115 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:38.589871 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:38.590873 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:38.591713 894 server_base.cc:1034] running on GCE node
I20240503 13:58:38.592087 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:38.592588 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:38.593729 894 hybrid_clock.cc:648] HybridClock initialized: now 1714744718593722 us; error 28 us; skew 500 ppm
I20240503 13:58:38.595500 894 webserver.cc:460] Webserver started at http://127.0.99.132:39507/ using document root <none> and password file <none>
I20240503 13:58:38.595978 894 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:38.596033 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:38.596185 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:38.597779 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e59e5d411bc046a28486330e3e4ffb66"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.598248 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e59e5d411bc046a28486330e3e4ffb66"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.601131 894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:58:38.603049 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.604004 894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:38.604101 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e59e5d411bc046a28486330e3e4ffb66"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.604177 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:38.638236 894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:38.638825 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:38.638974 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:38.639509 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:38.640316 894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:38.640370 894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.640427 894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:38.640467 894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.649469 894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:40971
I20240503 13:58:38.649502 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:40971 every 8 connection(s)
I20240503 13:58:38.650380 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:58:38.658519 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 894
I20240503 13:58:38.658560 1022 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:38.658633 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:58:38.658759 1022 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:38.659375 1022 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:38.660308 440 ts_manager.cc:194] Registered new tserver with Master: e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:38.660789 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40943
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 13:58:38.661013 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:56877
W20240503 13:58:38.763181 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:38.763440 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:38.763525 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:38.766320 1025 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:38.766386 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:38.766484 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:58:38.769697 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:38.770452 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:38.771497 1025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:38.779593 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:38.780277 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:38.781322 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:38.782102 1025 server_base.cc:1034] running on GCE node
I20240503 13:58:38.782449 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:38.782936 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:38.784071 1025 hybrid_clock.cc:648] HybridClock initialized: now 1714744718784060 us; error 37 us; skew 500 ppm
I20240503 13:58:38.785784 1025 webserver.cc:460] Webserver started at http://127.0.99.133:39727/ using document root <none> and password file <none>
I20240503 13:58:38.786265 1025 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:38.786322 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:38.786489 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:38.787952 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "83070528d9574a528c6e57fd30b33dcc"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.788389 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "83070528d9574a528c6e57fd30b33dcc"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.791325 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 13:58:38.793316 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.794214 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 13:58:38.794303 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "83070528d9574a528c6e57fd30b33dcc"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.794391 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:38.811200 1025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:38.811777 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:38.811945 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:38.812445 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:38.813256 1025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:38.813304 1025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.813390 1025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:38.813416 1025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.822144 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34511
I20240503 13:58:38.822165 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34511 every 8 connection(s)
I20240503 13:58:38.822999 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:58:38.826242 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 1025
I20240503 13:58:38.826354 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:58:38.828553 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40943
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 13:58:38.832468 1153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:38.832718 1153 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:38.833400 1153 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:38.834475 440 ts_manager.cc:194] Registered new tserver with Master: 83070528d9574a528c6e57fd30b33dcc (127.0.99.133:34511)
I20240503 13:58:38.835036 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:41641
W20240503 13:58:38.939836 1156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:38.940022 1156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:38.940079 1156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:38.942925 1156 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:38.942984 1156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:38.943084 1156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:58:38.946466 1156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.1156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:38.947264 1156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:38.948283 1156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:38.956146 1161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:38.956871 1164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:38.957664 1162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:38.958762 1156 server_base.cc:1034] running on GCE node
I20240503 13:58:38.959089 1156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:38.959578 1156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:38.960726 1156 hybrid_clock.cc:648] HybridClock initialized: now 1714744718960699 us; error 36 us; skew 500 ppm
I20240503 13:58:38.962471 1156 webserver.cc:460] Webserver started at http://127.0.99.134:33045/ using document root <none> and password file <none>
I20240503 13:58:38.962965 1156 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:38.963017 1156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:38.963198 1156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:38.964766 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "3be34dcec5ad4ec7860fe6145ac58163"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.965202 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "3be34dcec5ad4ec7860fe6145ac58163"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.968344 1156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:58:38.970327 1170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.971177 1156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:38.971297 1156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3be34dcec5ad4ec7860fe6145ac58163"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:38.971376 1156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:38.984961 1156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:38.985478 1156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:38.985621 1156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:38.986111 1156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:38.986918 1156 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:38.986965 1156 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.987021 1156 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:38.987048 1156 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:38.996603 1156 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:36679
I20240503 13:58:38.996642 1283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:36679 every 8 connection(s)
I20240503 13:58:38.997514 1156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:58:39.004161 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 1156
I20240503 13:58:39.004253 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 13:58:39.005553 1284 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:39.005748 1284 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:39.006187 1284 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:39.006258 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40943
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 13:58:39.007087 440 ts_manager.cc:194] Registered new tserver with Master: 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134:36679)
I20240503 13:58:39.007652 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:58791
W20240503 13:58:39.112138 1287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:39.112356 1287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:39.112416 1287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:39.115614 1287 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:39.115672 1287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:39.115787 1287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:58:39.120028 1287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.1287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:39.120872 1287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:39.121891 1287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:39.130065 1293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:39.130746 629 heartbeater.cc:503] Master 127.0.99.190:40943 was elected leader, sending a full tablet report...
W20240503 13:58:39.131013 1295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:39.131860 1292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:39.132751 1287 server_base.cc:1034] running on GCE node
I20240503 13:58:39.133070 1287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:39.133538 1287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:39.134680 1287 hybrid_clock.cc:648] HybridClock initialized: now 1714744719134665 us; error 34 us; skew 500 ppm
I20240503 13:58:39.136324 1287 webserver.cc:460] Webserver started at http://127.0.99.135:41317/ using document root <none> and password file <none>
I20240503 13:58:39.136826 1287 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:39.136881 1287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:39.137068 1287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:39.138628 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "ab254774096345abb0c1daf0f7afecac"
format_stamp: "Formatted at 2024-05-03 13:58:39 on dist-test-slave-snql"
I20240503 13:58:39.139042 1287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "ab254774096345abb0c1daf0f7afecac"
format_stamp: "Formatted at 2024-05-03 13:58:39 on dist-test-slave-snql"
I20240503 13:58:39.142185 1287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:39.144182 1301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:39.145051 1287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:39.145143 1287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ab254774096345abb0c1daf0f7afecac"
format_stamp: "Formatted at 2024-05-03 13:58:39 on dist-test-slave-snql"
I20240503 13:58:39.145206 1287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:39.157011 1287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:39.157536 1287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:39.157677 1287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:39.158128 1287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:39.158933 1287 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:39.158977 1287 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:39.159030 1287 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:39.159055 1287 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:39.167868 1287 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41905
I20240503 13:58:39.167892 1414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41905 every 8 connection(s)
I20240503 13:58:39.168749 1287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:58:39.171612 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 1287
I20240503 13:58:39.171701 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 13:58:39.173746 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40943
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 13:58:39.177538 1415 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:39.177737 1415 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:39.178200 1415 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:39.179162 440 ts_manager.cc:194] Registered new tserver with Master: ab254774096345abb0c1daf0f7afecac (127.0.99.135:41905)
I20240503 13:58:39.179726 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:52743
W20240503 13:58:39.277834 1418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:39.278025 1418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:39.278079 1418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:39.280970 1418 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:39.281026 1418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:39.281090 1418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 13:58:39.284546 1418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.1418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:39.285300 1418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:39.286291 1418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:39.294819 1423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:39.295753 1424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:39.296756 1426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:39.297569 1418 server_base.cc:1034] running on GCE node
I20240503 13:58:39.297945 1418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:39.298481 1418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:39.299629 1418 hybrid_clock.cc:648] HybridClock initialized: now 1714744719299615 us; error 28 us; skew 500 ppm
I20240503 13:58:39.301362 1418 webserver.cc:460] Webserver started at http://127.0.99.136:37193/ using document root <none> and password file <none>
I20240503 13:58:39.301887 1418 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:39.301940 1418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:39.302142 1418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:58:39.303856 1418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "06de56aefa784b4c8b7d5a3e7c5f9825"
format_stamp: "Formatted at 2024-05-03 13:58:39 on dist-test-slave-snql"
I20240503 13:58:39.304296 1418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "06de56aefa784b4c8b7d5a3e7c5f9825"
format_stamp: "Formatted at 2024-05-03 13:58:39 on dist-test-slave-snql"
I20240503 13:58:39.305481 760 heartbeater.cc:503] Master 127.0.99.190:40943 was elected leader, sending a full tablet report...
I20240503 13:58:39.307683 1418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:58:39.309676 1432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:39.310611 1418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:39.310730 1418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "06de56aefa784b4c8b7d5a3e7c5f9825"
format_stamp: "Formatted at 2024-05-03 13:58:39 on dist-test-slave-snql"
I20240503 13:58:39.310814 1418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:39.331859 1418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:39.332432 1418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:39.332584 1418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:39.333076 1418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:39.333906 1418 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:39.333971 1418 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:39.334048 1418 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:39.334105 1418 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:39.343014 1418 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:33227
I20240503 13:58:39.343060 1545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:33227 every 8 connection(s)
I20240503 13:58:39.343941 1418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:58:39.350078 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 1418
I20240503 13:58:39.350180 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 13:58:39.352245 1546 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:39.352455 1546 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:39.352980 1546 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:39.353919 441 ts_manager.cc:194] Registered new tserver with Master: 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136:33227)
I20240503 13:58:39.354409 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:47869
I20240503 13:58:39.364358 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 13:58:39.374490 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 13:58:39.374562 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 1025
I20240503 13:58:39.380295 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 1156
I20240503 13:58:39.387873 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 1287
I20240503 13:58:39.393262 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 1418
I20240503 13:58:39.473843 891 heartbeater.cc:503] Master 127.0.99.190:40943 was elected leader, sending a full tablet report...
I20240503 13:58:39.662945 1022 heartbeater.cc:503] Master 127.0.99.190:40943 was elected leader, sending a full tablet report...
I20240503 13:58:43.151149 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34534:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:43.172004 955 tablet_service.cc:1467] Processing CreateTablet for tablet 633e8264ce454183b2e0c0ad92f34629 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6fb6addecd294243a4c390358564540d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.171666 563 tablet_service.cc:1467] Processing CreateTablet for tablet a35914f78ffe423e91ce866785764b9c (DEFAULT_TABLE table=rebalance_test_table_0 [id=6fb6addecd294243a4c390358564540d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.171667 956 tablet_service.cc:1467] Processing CreateTablet for tablet a35914f78ffe423e91ce866785764b9c (DEFAULT_TABLE table=rebalance_test_table_0 [id=6fb6addecd294243a4c390358564540d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.171666 561 tablet_service.cc:1467] Processing CreateTablet for tablet 633e8264ce454183b2e0c0ad92f34629 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6fb6addecd294243a4c390358564540d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.172708 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35914f78ffe423e91ce866785764b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.172710 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35914f78ffe423e91ce866785764b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.172947 562 tablet_service.cc:1467] Processing CreateTablet for tablet 67ee1c64f81b492a9c83e7618878ff79 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6fb6addecd294243a4c390358564540d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.173617 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633e8264ce454183b2e0c0ad92f34629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.174497 825 tablet_service.cc:1467] Processing CreateTablet for tablet a35914f78ffe423e91ce866785764b9c (DEFAULT_TABLE table=rebalance_test_table_0 [id=6fb6addecd294243a4c390358564540d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.175308 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35914f78ffe423e91ce866785764b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.175798 694 tablet_service.cc:1467] Processing CreateTablet for tablet 67ee1c64f81b492a9c83e7618878ff79 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6fb6addecd294243a4c390358564540d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.176580 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ee1c64f81b492a9c83e7618878ff79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.177943 693 tablet_service.cc:1467] Processing CreateTablet for tablet 633e8264ce454183b2e0c0ad92f34629 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6fb6addecd294243a4c390358564540d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.178211 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633e8264ce454183b2e0c0ad92f34629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.174495 824 tablet_service.cc:1467] Processing CreateTablet for tablet 67ee1c64f81b492a9c83e7618878ff79 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6fb6addecd294243a4c390358564540d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.178740 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ee1c64f81b492a9c83e7618878ff79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.182439 1567 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576: Bootstrap starting.
I20240503 13:58:43.183104 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ee1c64f81b492a9c83e7618878ff79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.183228 1568 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4: Bootstrap starting.
I20240503 13:58:43.183320 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633e8264ce454183b2e0c0ad92f34629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.185364 1568 tablet_bootstrap.cc:654] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.185420 1566 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: Bootstrap starting.
I20240503 13:58:43.186187 1568 log.cc:824] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:43.186745 1567 tablet_bootstrap.cc:654] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.187228 1566 tablet_bootstrap.cc:654] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.187515 1567 log.cc:824] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:43.188053 1566 log.cc:824] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:43.188652 1571 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:43.189662 1568 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4: No bootstrap required, opened a new log
I20240503 13:58:43.189838 1568 ts_tablet_manager.cc:1367] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 13:58:43.190400 1571 tablet_bootstrap.cc:654] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.191053 1571 log.cc:824] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:43.191457 1566 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: No bootstrap required, opened a new log
I20240503 13:58:43.191648 1566 ts_tablet_manager.cc:1367] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 13:58:43.191660 1567 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576: No bootstrap required, opened a new log
I20240503 13:58:43.191812 1567 ts_tablet_manager.cc:1367] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 13:58:43.192459 1571 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:43.192586 1571 ts_tablet_manager.cc:1367] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:58:43.194383 1568 raft_consensus.cc:348] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.194581 1568 raft_consensus.cc:374] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.194629 1568 raft_consensus.cc:729] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.195047 1568 consensus_queue.cc:260] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.195628 1567 raft_consensus.cc:348] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.195840 1567 raft_consensus.cc:374] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.195920 1567 raft_consensus.cc:729] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.195956 1568 ts_tablet_manager.cc:1398] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 13:58:43.195921 1571 raft_consensus.cc:348] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.196110 1571 raft_consensus.cc:374] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.196184 1571 raft_consensus.cc:729] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.196313 1568 tablet_bootstrap.cc:492] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4: Bootstrap starting.
I20240503 13:58:43.196272 1566 raft_consensus.cc:348] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.196493 1566 raft_consensus.cc:374] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.196570 1566 raft_consensus.cc:729] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.196571 1571 consensus_queue.cc:260] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.197011 1566 consensus_queue.cc:260] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.197238 1568 tablet_bootstrap.cc:654] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.197412 1571 ts_tablet_manager.cc:1398] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 13:58:43.197399 1567 consensus_queue.cc:260] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.197803 1571 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:43.198278 1567 ts_tablet_manager.cc:1398] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 13:58:43.198606 1567 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576: Bootstrap starting.
I20240503 13:58:43.199487 1571 tablet_bootstrap.cc:654] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.199738 1567 tablet_bootstrap.cc:654] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.200228 1566 ts_tablet_manager.cc:1398] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: Time spent starting tablet: real 0.008s user 0.001s sys 0.003s
I20240503 13:58:43.200575 1566 tablet_bootstrap.cc:492] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66: Bootstrap starting.
I20240503 13:58:43.200870 1571 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:43.200951 1571 ts_tablet_manager.cc:1367] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:43.201102 1567 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576: No bootstrap required, opened a new log
I20240503 13:58:43.201206 1567 ts_tablet_manager.cc:1367] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:43.201485 1571 raft_consensus.cc:348] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.201584 1571 raft_consensus.cc:374] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.201622 1571 raft_consensus.cc:729] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.201638 1566 tablet_bootstrap.cc:654] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.201740 1571 consensus_queue.cc:260] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.201910 1567 raft_consensus.cc:348] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.202018 1571 ts_tablet_manager.cc:1398] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:43.202134 1567 raft_consensus.cc:374] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.202164 1571 tablet_bootstrap.cc:492] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:43.202220 1567 raft_consensus.cc:729] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.202396 1567 consensus_queue.cc:260] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.202739 1567 ts_tablet_manager.cc:1398] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:43.203022 1571 tablet_bootstrap.cc:654] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.203876 1566 tablet_bootstrap.cc:492] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66: No bootstrap required, opened a new log
I20240503 13:58:43.203963 1566 ts_tablet_manager.cc:1367] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:58:43.203987 1568 tablet_bootstrap.cc:492] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4: No bootstrap required, opened a new log
I20240503 13:58:43.204061 1568 ts_tablet_manager.cc:1367] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:58:43.204434 1566 raft_consensus.cc:348] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:43.204525 1566 raft_consensus.cc:374] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.204553 1566 raft_consensus.cc:729] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.204500 1568 raft_consensus.cc:348] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:43.204593 1568 raft_consensus.cc:374] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.204650 1568 raft_consensus.cc:729] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.204699 1566 consensus_queue.cc:260] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:43.204874 1568 consensus_queue.cc:260] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:43.205018 1566 ts_tablet_manager.cc:1398] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:58:43.205273 1568 ts_tablet_manager.cc:1398] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:43.205901 1571 tablet_bootstrap.cc:492] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:43.205979 1571 ts_tablet_manager.cc:1367] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:43.206404 1571 raft_consensus.cc:348] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:43.206486 1571 raft_consensus.cc:374] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.206516 1571 raft_consensus.cc:729] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.206630 1571 consensus_queue.cc:260] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:43.206849 1571 ts_tablet_manager.cc:1398] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 13:58:43.215211 892 tablet.cc:2369] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:58:43.279269 1575 raft_consensus.cc:482] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:43.279459 1575 raft_consensus.cc:504] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.280645 1575 leader_election.cc:288] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:58:43.285506 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ee1c64f81b492a9c83e7618878ff79" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" is_pre_election: true
I20240503 13:58:43.285532 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ee1c64f81b492a9c83e7618878ff79" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:43.285789 714 raft_consensus.cc:2454] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 0.
I20240503 13:58:43.285807 845 raft_consensus.cc:2454] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 0.
I20240503 13:58:43.286213 519 leader_election.cc:302] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, 769439c9b5d3418eb0ebd0287954e6cc; no voters:
I20240503 13:58:43.286427 1575 raft_consensus.cc:2781] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:43.286484 1575 raft_consensus.cc:482] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:43.286517 1575 raft_consensus.cc:3037] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
W20240503 13:58:43.294648 761 tablet.cc:2369] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:58:43.337813 1578 raft_consensus.cc:482] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:43.338037 1578 raft_consensus.cc:504] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.339196 1578 leader_election.cc:288] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:58:43.340981 1575 raft_consensus.cc:504] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.341377 1575 leader_election.cc:288] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 election: Requested vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:58:43.341871 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ee1c64f81b492a9c83e7618878ff79" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:43.341951 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ee1c64f81b492a9c83e7618878ff79" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
I20240503 13:58:43.342000 845 raft_consensus.cc:3037] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:43.342090 714 raft_consensus.cc:3037] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:43.343286 845 raft_consensus.cc:2454] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 1.
I20240503 13:58:43.343371 714 raft_consensus.cc:2454] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 1.
I20240503 13:58:43.343648 518 leader_election.cc:302] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:43.343906 1575 raft_consensus.cc:2781] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:43.343979 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633e8264ce454183b2e0c0ad92f34629" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" is_pre_election: true
I20240503 13:58:43.344099 714 raft_consensus.cc:2454] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 0.
I20240503 13:58:43.344141 1575 raft_consensus.cc:686] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 LEADER]: Becoming Leader. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Running, Role: LEADER
I20240503 13:58:43.344453 912 leader_election.cc:302] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:43.344481 1575 consensus_queue.cc:237] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.344714 1578 raft_consensus.cc:2781] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:43.344769 1578 raft_consensus.cc:482] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:43.344805 1578 raft_consensus.cc:3037] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:43.345820 1578 raft_consensus.cc:504] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.346226 1578 leader_election.cc:288] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Requested vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:58:43.346414 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633e8264ce454183b2e0c0ad92f34629" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc" is_pre_election: true
I20240503 13:58:43.346606 583 raft_consensus.cc:2454] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 0.
I20240503 13:58:43.347038 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633e8264ce454183b2e0c0ad92f34629" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc"
I20240503 13:58:43.347136 583 raft_consensus.cc:3037] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:43.347343 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633e8264ce454183b2e0c0ad92f34629" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
I20240503 13:58:43.347455 714 raft_consensus.cc:3037] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:43.348614 441 catalog_manager.cc:5484] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129). New cstate: current_term: 1 leader_uuid: "769439c9b5d3418eb0ebd0287954e6cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: HEALTHY } } }
W20240503 13:58:43.372246 630 tablet.cc:2369] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:58:43.404281 1023 tablet.cc:2369] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:58:43.438457 583 raft_consensus.cc:2454] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:43.438457 714 raft_consensus.cc:2454] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:43.439050 909 leader_election.cc:302] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:43.439309 1578 raft_consensus.cc:2781] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:43.439582 1578 raft_consensus.cc:686] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Becoming Leader. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Running, Role: LEADER
I20240503 13:58:43.439987 1578 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:43.443645 441 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:43.531149 1588 raft_consensus.cc:482] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:43.531327 1588 raft_consensus.cc:504] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:43.531867 1588 leader_election.cc:288] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:43.532382 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35914f78ffe423e91ce866785764b9c" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:43.532529 845 raft_consensus.cc:2454] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 0.
I20240503 13:58:43.532842 518 leader_election.cc:302] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:43.533022 1588 raft_consensus.cc:2781] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:43.533097 1588 raft_consensus.cc:482] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:43.533151 1588 raft_consensus.cc:3037] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:43.535775 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35914f78ffe423e91ce866785764b9c" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66" is_pre_election: true
I20240503 13:58:43.535984 976 raft_consensus.cc:2454] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 0.
I20240503 13:58:43.540880 1588 raft_consensus.cc:504] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:43.541249 1588 leader_election.cc:288] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 election: Requested vote from peers e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:43.541625 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35914f78ffe423e91ce866785764b9c" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
I20240503 13:58:43.541662 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35914f78ffe423e91ce866785764b9c" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:43.541760 976 raft_consensus.cc:3037] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:43.541777 845 raft_consensus.cc:3037] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:43.542583 845 raft_consensus.cc:2454] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 1.
I20240503 13:58:43.542908 518 leader_election.cc:302] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:43.543087 1588 raft_consensus.cc:2781] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:43.543071 976 raft_consensus.cc:2454] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 1.
I20240503 13:58:43.543165 1588 raft_consensus.cc:686] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 1 LEADER]: Becoming Leader. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Running, Role: LEADER
I20240503 13:58:43.543327 1588 consensus_queue.cc:237] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:43.545094 441 catalog_manager.cc:5484] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129). New cstate: current_term: 1 leader_uuid: "769439c9b5d3418eb0ebd0287954e6cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:43.735967 1593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:43.736215 1593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:43.739310 1593 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 13:58:43.819978 845 raft_consensus.cc:1261] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 769439c9b5d3418eb0ebd0287954e6cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:43.819965 714 raft_consensus.cc:1261] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer 769439c9b5d3418eb0ebd0287954e6cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:43.820627 1575 consensus_queue.cc:1035] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:43.820768 1588 consensus_queue.cc:1035] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:43.822167 976 raft_consensus.cc:1261] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer 769439c9b5d3418eb0ebd0287954e6cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:43.823418 583 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:43.824035 1578 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:43.825139 844 raft_consensus.cc:1261] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 769439c9b5d3418eb0ebd0287954e6cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:43.825592 1575 consensus_queue.cc:1035] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:43.826687 714 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:43.827798 1588 consensus_queue.cc:1035] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:43.827956 1578 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:43.829051 1624 mvcc.cc:205] Tried to move back new op lower bound from 7023594388773773312 to 7023594387213750272. Current Snapshot: MvccSnapshot[applied={T|T < 7023594388773773312}]
I20240503 13:58:43.831847 1622 mvcc.cc:205] Tried to move back new op lower bound from 7023594388773773312 to 7023594387213750272. Current Snapshot: MvccSnapshot[applied={T|T < 7023594388773773312}]
I20240503 13:58:43.861763 1644 mvcc.cc:205] Tried to move back new op lower bound from 7023594388789821440 to 7023594387213750272. Current Snapshot: MvccSnapshot[applied={T|T < 7023594388789821440}]
I20240503 13:58:43.859172 1641 mvcc.cc:205] Tried to move back new op lower bound from 7023594388759330816 to 7023594386822074368. Current Snapshot: MvccSnapshot[applied={T|T < 7023594388759330816}]
INSERT report
rows total: 2000
time total: 114.724 ms
time per row: 0.0573622 ms
I20240503 13:58:43.896647 437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34534:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:43.902395 693 tablet_service.cc:1467] Processing CreateTablet for tablet 2f90e135e32343dfb01a1e435e87bd3e (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.902731 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f90e135e32343dfb01a1e435e87bd3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.903211 956 tablet_service.cc:1467] Processing CreateTablet for tablet 2f90e135e32343dfb01a1e435e87bd3e (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.903293 955 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc710cdf7be47dcbb95ccc49e5beabf (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.903486 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f90e135e32343dfb01a1e435e87bd3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.903968 563 tablet_service.cc:1467] Processing CreateTablet for tablet eb3d68a0d39042c98d02c9a93307bb88 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.904249 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3d68a0d39042c98d02c9a93307bb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.905545 1657 tablet_bootstrap.cc:492] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66: Bootstrap starting.
I20240503 13:58:43.906167 694 tablet_service.cc:1467] Processing CreateTablet for tablet eb3d68a0d39042c98d02c9a93307bb88 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.906411 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3d68a0d39042c98d02c9a93307bb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.906447 1657 tablet_bootstrap.cc:654] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.906929 825 tablet_service.cc:1467] Processing CreateTablet for tablet 2f90e135e32343dfb01a1e435e87bd3e (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.907191 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f90e135e32343dfb01a1e435e87bd3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.907193 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc710cdf7be47dcbb95ccc49e5beabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.907496 823 tablet_service.cc:1467] Processing CreateTablet for tablet eb3d68a0d39042c98d02c9a93307bb88 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.907706 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3d68a0d39042c98d02c9a93307bb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.909529 824 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc710cdf7be47dcbb95ccc49e5beabf (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.909708 1659 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:43.909785 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc710cdf7be47dcbb95ccc49e5beabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.909811 1657 tablet_bootstrap.cc:492] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66: No bootstrap required, opened a new log
I20240503 13:58:43.909896 1657 ts_tablet_manager.cc:1367] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:43.910246 1657 raft_consensus.cc:348] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.910379 1657 raft_consensus.cc:374] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.910403 1657 raft_consensus.cc:729] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.910516 1657 consensus_queue.cc:260] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.910598 1659 tablet_bootstrap.cc:654] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.910766 1657 ts_tablet_manager.cc:1398] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:43.910921 1657 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66: Bootstrap starting.
I20240503 13:58:43.911844 693 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc710cdf7be47dcbb95ccc49e5beabf (DEFAULT_TABLE table=rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:43.912077 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc710cdf7be47dcbb95ccc49e5beabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:43.912431 1657 tablet_bootstrap.cc:654] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.913888 1657 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66: No bootstrap required, opened a new log
I20240503 13:58:43.913990 1659 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:43.914031 1657 ts_tablet_manager.cc:1367] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:43.914098 1656 tablet_bootstrap.cc:492] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576: Bootstrap starting.
I20240503 13:58:43.914180 1659 ts_tablet_manager.cc:1367] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:43.914500 1657 raft_consensus.cc:348] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.914618 1657 raft_consensus.cc:374] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.914650 1657 raft_consensus.cc:729] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.914692 1659 raft_consensus.cc:348] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:43.914806 1659 raft_consensus.cc:374] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.914785 1657 consensus_queue.cc:260] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.914881 1659 raft_consensus.cc:729] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.915086 1657 ts_tablet_manager.cc:1398] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:43.915122 1656 tablet_bootstrap.cc:654] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.915455 1660 tablet_bootstrap.cc:492] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4: Bootstrap starting.
I20240503 13:58:43.915333 1659 consensus_queue.cc:260] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:43.916420 1660 tablet_bootstrap.cc:654] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.916500 1656 tablet_bootstrap.cc:492] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576: No bootstrap required, opened a new log
I20240503 13:58:43.916571 1656 ts_tablet_manager.cc:1367] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:43.916967 1659 ts_tablet_manager.cc:1398] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:43.917011 1656 raft_consensus.cc:348] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.917114 1656 raft_consensus.cc:374] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.917155 1656 raft_consensus.cc:729] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.917279 1656 consensus_queue.cc:260] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.917498 1656 ts_tablet_manager.cc:1398] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:43.917693 1656 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576: Bootstrap starting.
I20240503 13:58:43.918488 1656 tablet_bootstrap.cc:654] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.919485 1656 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576: No bootstrap required, opened a new log
I20240503 13:58:43.919709 1656 ts_tablet_manager.cc:1367] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:43.919741 1660 tablet_bootstrap.cc:492] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4: No bootstrap required, opened a new log
I20240503 13:58:43.919893 1660 ts_tablet_manager.cc:1367] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:58:43.920179 1656 raft_consensus.cc:348] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:43.920313 1656 raft_consensus.cc:374] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.920359 1656 raft_consensus.cc:729] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.920501 1656 consensus_queue.cc:260] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:43.920766 1656 ts_tablet_manager.cc:1398] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:43.920938 1656 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576: Bootstrap starting.
I20240503 13:58:43.921027 1660 raft_consensus.cc:348] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.921144 1660 raft_consensus.cc:374] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.921203 1660 raft_consensus.cc:729] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.921399 1660 consensus_queue.cc:260] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.921694 1656 tablet_bootstrap.cc:654] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.921989 1660 ts_tablet_manager.cc:1398] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:43.922217 1660 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4: Bootstrap starting.
I20240503 13:58:43.923105 1660 tablet_bootstrap.cc:654] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.923893 1656 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576: No bootstrap required, opened a new log
I20240503 13:58:43.923990 1656 ts_tablet_manager.cc:1367] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:58:43.924252 1660 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4: No bootstrap required, opened a new log
I20240503 13:58:43.924345 1660 ts_tablet_manager.cc:1367] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:43.924738 1656 raft_consensus.cc:348] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.924741 1660 raft_consensus.cc:348] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:43.924834 1660 raft_consensus.cc:374] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.924834 1656 raft_consensus.cc:374] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.924865 1660 raft_consensus.cc:729] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.924865 1656 raft_consensus.cc:729] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.925002 1660 consensus_queue.cc:260] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:43.925034 1656 consensus_queue.cc:260] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.925278 1660 ts_tablet_manager.cc:1398] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:43.925446 1660 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: Bootstrap starting.
I20240503 13:58:43.925611 1656 ts_tablet_manager.cc:1398] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:58:43.926113 1660 tablet_bootstrap.cc:654] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:43.927270 1660 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: No bootstrap required, opened a new log
I20240503 13:58:43.927359 1660 ts_tablet_manager.cc:1367] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:43.927783 1660 raft_consensus.cc:348] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.927985 1660 raft_consensus.cc:374] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:43.928037 1660 raft_consensus.cc:729] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Initialized, Role: FOLLOWER
I20240503 13:58:43.928153 1660 consensus_queue.cc:260] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.928381 1660 ts_tablet_manager.cc:1398] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:43.985615 1627 raft_consensus.cc:482] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:43.985745 1627 raft_consensus.cc:504] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:43.986763 1627 leader_election.cc:288] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:43.991415 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f90e135e32343dfb01a1e435e87bd3e" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:43.991569 845 raft_consensus.cc:2454] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 0.
I20240503 13:58:43.991814 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f90e135e32343dfb01a1e435e87bd3e" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66" is_pre_election: true
I20240503 13:58:43.991948 976 raft_consensus.cc:2454] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 0.
I20240503 13:58:43.991948 649 leader_election.cc:302] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:43.992189 1627 raft_consensus.cc:2781] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:43.992252 1627 raft_consensus.cc:482] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:43.992292 1627 raft_consensus.cc:3037] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.001374 1627 raft_consensus.cc:504] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.001757 1627 leader_election.cc:288] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 election: Requested vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:44.002097 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f90e135e32343dfb01a1e435e87bd3e" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:44.002141 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f90e135e32343dfb01a1e435e87bd3e" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
I20240503 13:58:44.002251 976 raft_consensus.cc:3037] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.002224 845 raft_consensus.cc:3037] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.003211 976 raft_consensus.cc:2454] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 1.
I20240503 13:58:44.003209 845 raft_consensus.cc:2454] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 1.
I20240503 13:58:44.003535 647 leader_election.cc:302] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:44.003711 1627 raft_consensus.cc:2781] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:44.003835 1627 raft_consensus.cc:686] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Becoming Leader. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Running, Role: LEADER
I20240503 13:58:44.004236 1627 consensus_queue.cc:237] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.006791 439 catalog_manager.cc:5484] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:44.014772 1632 raft_consensus.cc:482] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:44.014863 1632 raft_consensus.cc:504] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.015744 1632 leader_election.cc:288] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:44.019186 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc710cdf7be47dcbb95ccc49e5beabf" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66" is_pre_election: true
I20240503 13:58:44.019351 976 raft_consensus.cc:2454] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58fb372531d42a3bc405510bf5bced4 in term 0.
I20240503 13:58:44.019620 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc710cdf7be47dcbb95ccc49e5beabf" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" is_pre_election: true
I20240503 13:58:44.019645 778 leader_election.cc:302] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e59e5d411bc046a28486330e3e4ffb66, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:44.019752 713 raft_consensus.cc:2454] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58fb372531d42a3bc405510bf5bced4 in term 0.
I20240503 13:58:44.019902 1632 raft_consensus.cc:2781] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:44.019968 1632 raft_consensus.cc:482] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:44.020023 1632 raft_consensus.cc:3037] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.078769 1632 raft_consensus.cc:504] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.079433 1632 leader_election.cc:288] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 1 election: Requested vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:44.079923 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc710cdf7be47dcbb95ccc49e5beabf" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
I20240503 13:58:44.080051 713 raft_consensus.cc:3037] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.080000 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc710cdf7be47dcbb95ccc49e5beabf" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
I20240503 13:58:44.080163 976 raft_consensus.cc:3037] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.081355 713 raft_consensus.cc:2454] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58fb372531d42a3bc405510bf5bced4 in term 1.
I20240503 13:58:44.081432 976 raft_consensus.cc:2454] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58fb372531d42a3bc405510bf5bced4 in term 1.
I20240503 13:58:44.081735 781 leader_election.cc:302] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:44.081949 1632 raft_consensus.cc:2781] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:44.082259 1632 raft_consensus.cc:686] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Becoming Leader. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Running, Role: LEADER
I20240503 13:58:44.082679 1632 consensus_queue.cc:237] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.085322 439 catalog_manager.cc:5484] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58fb372531d42a3bc405510bf5bced4 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:44.159883 1632 raft_consensus.cc:482] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:44.160120 1632 raft_consensus.cc:504] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.160998 1632 leader_election.cc:288] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:58:44.161491 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3d68a0d39042c98d02c9a93307bb88" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" is_pre_election: true
I20240503 13:58:44.161651 713 raft_consensus.cc:2454] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58fb372531d42a3bc405510bf5bced4 in term 0.
I20240503 13:58:44.161947 781 leader_election.cc:302] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:44.162110 1632 raft_consensus.cc:2781] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:44.162169 1632 raft_consensus.cc:482] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:44.162215 1632 raft_consensus.cc:3037] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.164758 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3d68a0d39042c98d02c9a93307bb88" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc" is_pre_election: true
I20240503 13:58:44.164913 583 raft_consensus.cc:2454] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58fb372531d42a3bc405510bf5bced4 in term 0.
I20240503 13:58:44.231748 1632 raft_consensus.cc:504] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.232179 1632 leader_election.cc:288] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 1 election: Requested vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:58:44.232537 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3d68a0d39042c98d02c9a93307bb88" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc"
I20240503 13:58:44.232673 583 raft_consensus.cc:3037] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.232652 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3d68a0d39042c98d02c9a93307bb88" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
I20240503 13:58:44.232767 713 raft_consensus.cc:3037] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.233701 713 raft_consensus.cc:2454] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58fb372531d42a3bc405510bf5bced4 in term 1.
I20240503 13:58:44.233701 583 raft_consensus.cc:2454] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58fb372531d42a3bc405510bf5bced4 in term 1.
I20240503 13:58:44.234009 781 leader_election.cc:302] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:44.234202 1632 raft_consensus.cc:2781] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:44.234323 1632 raft_consensus.cc:686] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Becoming Leader. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Running, Role: LEADER
I20240503 13:58:44.234478 1632 consensus_queue.cc:237] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.236552 439 catalog_manager.cc:5484] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f58fb372531d42a3bc405510bf5bced4 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:44.393586 1627 consensus_queue.cc:1035] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:58:44.395174 1679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:44.395489 1679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 13:58:44.397882 1627 consensus_queue.cc:1035] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 13:58:44.399904 1679 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 13:58:44.487637 583 raft_consensus.cc:1261] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:44.487725 976 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:44.488335 713 raft_consensus.cc:1261] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:44.488420 714 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:44.488899 1676 consensus_queue.cc:1035] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:44.489320 1676 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:44.489583 1676 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:44.492110 1632 consensus_queue.cc:1035] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.297 ms
time per row: 0.0521484 ms
I20240503 13:58:44.547422 439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34534:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:44.552955 693 tablet_service.cc:1467] Processing CreateTablet for tablet 857e4baba1984dc4af9c81ee2baf2fba (DEFAULT_TABLE table=rebalance_test_table_2 [id=266f7578a0d345ffa1be58fe90375e59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:44.553014 563 tablet_service.cc:1467] Processing CreateTablet for tablet 857e4baba1984dc4af9c81ee2baf2fba (DEFAULT_TABLE table=rebalance_test_table_2 [id=266f7578a0d345ffa1be58fe90375e59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:44.553333 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857e4baba1984dc4af9c81ee2baf2fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:44.553334 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857e4baba1984dc4af9c81ee2baf2fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:44.553800 561 tablet_service.cc:1467] Processing CreateTablet for tablet 920e110f079f43dc89feb833979fdbc6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=266f7578a0d345ffa1be58fe90375e59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:44.554013 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920e110f079f43dc89feb833979fdbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:44.553972 956 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd93f79958e489b9028f5123c600f74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=266f7578a0d345ffa1be58fe90375e59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:44.554224 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd93f79958e489b9028f5123c600f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:44.555495 954 tablet_service.cc:1467] Processing CreateTablet for tablet 920e110f079f43dc89feb833979fdbc6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=266f7578a0d345ffa1be58fe90375e59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:44.555729 954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920e110f079f43dc89feb833979fdbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:44.555775 955 tablet_service.cc:1467] Processing CreateTablet for tablet 857e4baba1984dc4af9c81ee2baf2fba (DEFAULT_TABLE table=rebalance_test_table_2 [id=266f7578a0d345ffa1be58fe90375e59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:44.555994 955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857e4baba1984dc4af9c81ee2baf2fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:44.556051 1739 tablet_bootstrap.cc:492] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:44.556927 1739 tablet_bootstrap.cc:654] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:44.557469 1741 tablet_bootstrap.cc:492] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66: Bootstrap starting.
I20240503 13:58:44.557658 562 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd93f79958e489b9028f5123c600f74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=266f7578a0d345ffa1be58fe90375e59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:44.557859 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd93f79958e489b9028f5123c600f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:44.558252 824 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd93f79958e489b9028f5123c600f74 (DEFAULT_TABLE table=rebalance_test_table_2 [id=266f7578a0d345ffa1be58fe90375e59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:44.558351 1741 tablet_bootstrap.cc:654] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:44.558528 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd93f79958e489b9028f5123c600f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:44.559026 1739 tablet_bootstrap.cc:492] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:44.559099 1739 ts_tablet_manager.cc:1367] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:44.559528 1739 raft_consensus.cc:348] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.559652 1739 raft_consensus.cc:374] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:44.559685 1739 raft_consensus.cc:729] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:44.559823 1740 tablet_bootstrap.cc:492] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576: Bootstrap starting.
I20240503 13:58:44.559809 1739 consensus_queue.cc:260] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.560081 1739 ts_tablet_manager.cc:1398] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:44.560236 1739 tablet_bootstrap.cc:492] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:44.560808 1740 tablet_bootstrap.cc:654] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:44.561036 1741 tablet_bootstrap.cc:492] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66: No bootstrap required, opened a new log
I20240503 13:58:44.561043 1739 tablet_bootstrap.cc:654] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:44.561115 1741 ts_tablet_manager.cc:1367] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:58:44.561589 1744 tablet_bootstrap.cc:492] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4: Bootstrap starting.
I20240503 13:58:44.561558 1741 raft_consensus.cc:348] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.561651 1741 raft_consensus.cc:374] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:44.561691 1741 raft_consensus.cc:729] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Initialized, Role: FOLLOWER
I20240503 13:58:44.561846 1741 consensus_queue.cc:260] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.561962 1739 tablet_bootstrap.cc:492] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:44.562043 1739 ts_tablet_manager.cc:1367] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:44.562387 1739 raft_consensus.cc:348] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.562477 1739 raft_consensus.cc:374] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:44.562505 1739 raft_consensus.cc:729] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:44.562611 1739 consensus_queue.cc:260] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.562846 1744 tablet_bootstrap.cc:654] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:44.562901 1739 ts_tablet_manager.cc:1398] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:44.563068 1739 tablet_bootstrap.cc:492] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:44.563689 693 tablet_service.cc:1467] Processing CreateTablet for tablet 920e110f079f43dc89feb833979fdbc6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=266f7578a0d345ffa1be58fe90375e59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:44.563867 1739 tablet_bootstrap.cc:654] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:44.563936 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920e110f079f43dc89feb833979fdbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:44.564007 1741 ts_tablet_manager.cc:1398] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:44.565061 1740 tablet_bootstrap.cc:492] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576: No bootstrap required, opened a new log
I20240503 13:58:44.565079 1739 tablet_bootstrap.cc:492] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:44.565136 1739 ts_tablet_manager.cc:1367] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:44.565140 1740 ts_tablet_manager.cc:1367] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:44.565521 1739 raft_consensus.cc:348] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.565629 1739 raft_consensus.cc:374] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:44.565665 1739 raft_consensus.cc:729] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:44.565629 1740 raft_consensus.cc:348] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.565721 1740 raft_consensus.cc:374] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:44.565752 1740 raft_consensus.cc:729] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Initialized, Role: FOLLOWER
I20240503 13:58:44.565774 1739 consensus_queue.cc:260] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.565925 1740 consensus_queue.cc:260] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.566042 1739 ts_tablet_manager.cc:1398] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:44.566262 1740 ts_tablet_manager.cc:1398] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:44.566285 1741 tablet_bootstrap.cc:492] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66: Bootstrap starting.
I20240503 13:58:44.566892 1740 tablet_bootstrap.cc:492] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576: Bootstrap starting.
I20240503 13:58:44.566987 1744 tablet_bootstrap.cc:492] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4: No bootstrap required, opened a new log
I20240503 13:58:44.567070 1744 ts_tablet_manager.cc:1367] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 13:58:44.567596 1744 raft_consensus.cc:348] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.567731 1740 tablet_bootstrap.cc:654] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:44.567727 1744 raft_consensus.cc:374] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:44.567762 1744 raft_consensus.cc:729] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Initialized, Role: FOLLOWER
I20240503 13:58:44.567915 1744 consensus_queue.cc:260] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.568240 1744 ts_tablet_manager.cc:1398] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 13:58:44.569319 1740 tablet_bootstrap.cc:492] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576: No bootstrap required, opened a new log
I20240503 13:58:44.569407 1740 ts_tablet_manager.cc:1367] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:44.569501 1741 tablet_bootstrap.cc:654] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:44.569886 1740 raft_consensus.cc:348] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.569988 1740 raft_consensus.cc:374] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:44.570035 1740 raft_consensus.cc:729] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Initialized, Role: FOLLOWER
I20240503 13:58:44.570155 1740 consensus_queue.cc:260] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.570426 1740 ts_tablet_manager.cc:1398] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:44.571208 1741 tablet_bootstrap.cc:492] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66: No bootstrap required, opened a new log
I20240503 13:58:44.571347 1741 ts_tablet_manager.cc:1367] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:44.571797 1741 raft_consensus.cc:348] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.571887 1741 raft_consensus.cc:374] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:44.571919 1741 raft_consensus.cc:729] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Initialized, Role: FOLLOWER
I20240503 13:58:44.572049 1741 consensus_queue.cc:260] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.572314 1741 ts_tablet_manager.cc:1398] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:44.572530 1741 tablet_bootstrap.cc:492] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66: Bootstrap starting.
I20240503 13:58:44.573324 1741 tablet_bootstrap.cc:654] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:44.574517 1741 tablet_bootstrap.cc:492] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66: No bootstrap required, opened a new log
I20240503 13:58:44.574609 1741 ts_tablet_manager.cc:1367] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:44.574987 1741 raft_consensus.cc:348] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.575079 1741 raft_consensus.cc:374] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:44.575099 1741 raft_consensus.cc:729] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Initialized, Role: FOLLOWER
I20240503 13:58:44.575212 1741 consensus_queue.cc:260] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.575505 1741 ts_tablet_manager.cc:1398] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:44.606670 1575 raft_consensus.cc:482] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:44.606809 1575 raft_consensus.cc:504] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.607260 1575 leader_election.cc:288] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:44.607692 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd93f79958e489b9028f5123c600f74" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:44.607702 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd93f79958e489b9028f5123c600f74" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66" is_pre_election: true
I20240503 13:58:44.607820 845 raft_consensus.cc:2454] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 0.
I20240503 13:58:44.607828 976 raft_consensus.cc:2454] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 0.
I20240503 13:58:44.608170 516 leader_election.cc:302] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:44.608342 1575 raft_consensus.cc:2781] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:44.608399 1575 raft_consensus.cc:482] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:44.608424 1575 raft_consensus.cc:3037] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.671561 1630 raft_consensus.cc:482] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:44.671710 1630 raft_consensus.cc:504] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.672163 1630 leader_election.cc:288] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:58:44.672595 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920e110f079f43dc89feb833979fdbc6" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc" is_pre_election: true
I20240503 13:58:44.672729 583 raft_consensus.cc:2454] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 0.
I20240503 13:58:44.672922 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920e110f079f43dc89feb833979fdbc6" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" is_pre_election: true
I20240503 13:58:44.673063 713 raft_consensus.cc:2454] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 0.
I20240503 13:58:44.673072 909 leader_election.cc:302] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:44.673233 1630 raft_consensus.cc:2781] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:44.673290 1630 raft_consensus.cc:482] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:44.673332 1630 raft_consensus.cc:3037] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.687453 1575 raft_consensus.cc:504] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.687453 1630 raft_consensus.cc:504] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.687973 1630 leader_election.cc:288] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Requested vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:58:44.687970 1575 leader_election.cc:288] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 election: Requested vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:44.688383 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920e110f079f43dc89feb833979fdbc6" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc"
I20240503 13:58:44.688505 583 raft_consensus.cc:3037] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.688522 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd93f79958e489b9028f5123c600f74" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:44.688665 845 raft_consensus.cc:3037] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.689306 583 raft_consensus.cc:2454] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:44.689654 909 leader_election.cc:302] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:44.689677 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd93f79958e489b9028f5123c600f74" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
I20240503 13:58:44.689769 976 raft_consensus.cc:3037] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.689877 845 raft_consensus.cc:2454] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 1.
I20240503 13:58:44.689905 1630 raft_consensus.cc:2781] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:44.690115 1630 raft_consensus.cc:686] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Becoming Leader. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Running, Role: LEADER
I20240503 13:58:44.690256 518 leader_election.cc:302] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:44.690253 1630 consensus_queue.cc:237] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:44.690448 1575 raft_consensus.cc:2781] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:44.690522 1575 raft_consensus.cc:686] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 LEADER]: Becoming Leader. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Running, Role: LEADER
I20240503 13:58:44.690709 976 raft_consensus.cc:2454] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 1.
I20240503 13:58:44.690665 1575 consensus_queue.cc:237] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.691401 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "920e110f079f43dc89feb833979fdbc6" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
I20240503 13:58:44.691540 713 raft_consensus.cc:3037] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.692531 713 raft_consensus.cc:2454] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:44.692725 439 catalog_manager.cc:5484] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129). New cstate: current_term: 1 leader_uuid: "769439c9b5d3418eb0ebd0287954e6cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:44.693559 436 catalog_manager.cc:5484] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:44.797734 1753 raft_consensus.cc:482] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:44.797859 1753 raft_consensus.cc:504] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.798298 1753 leader_election.cc:288] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:58:44.798671 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e4baba1984dc4af9c81ee2baf2fba" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc" is_pre_election: true
I20240503 13:58:44.798679 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e4baba1984dc4af9c81ee2baf2fba" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" is_pre_election: true
I20240503 13:58:44.798800 713 raft_consensus.cc:2454] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 0.
I20240503 13:58:44.798826 583 raft_consensus.cc:2454] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 0.
I20240503 13:58:44.799111 909 leader_election.cc:302] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:44.799302 1753 raft_consensus.cc:2781] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:44.799355 1753 raft_consensus.cc:482] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:44.799381 1753 raft_consensus.cc:3037] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.860807 1753 raft_consensus.cc:504] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.861303 1753 leader_election.cc:288] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Requested vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:58:44.861785 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e4baba1984dc4af9c81ee2baf2fba" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc"
I20240503 13:58:44.861812 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857e4baba1984dc4af9c81ee2baf2fba" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
I20240503 13:58:44.861920 583 raft_consensus.cc:3037] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.861989 713 raft_consensus.cc:3037] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:44.862754 583 raft_consensus.cc:2454] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:44.863098 909 leader_election.cc:302] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:44.863166 713 raft_consensus.cc:2454] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:44.863318 1753 raft_consensus.cc:2781] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:44.863399 1753 raft_consensus.cc:686] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Becoming Leader. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Running, Role: LEADER
I20240503 13:58:44.863536 1753 consensus_queue.cc:237] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:44.865144 436 catalog_manager.cc:5484] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } }
W20240503 13:58:45.026762 1757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:45.026989 1757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:45.030197 1757 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 13:58:45.080250 1752 consensus_queue.cc:1035] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:45.083786 1575 consensus_queue.cc:1035] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:45.109823 713 raft_consensus.cc:1261] T 920e110f079f43dc89feb833979fdbc6 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:45.110452 1630 consensus_queue.cc:1035] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:45.111057 583 raft_consensus.cc:1261] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:45.111685 1630 consensus_queue.cc:1035] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:45.136024 713 raft_consensus.cc:1261] T 857e4baba1984dc4af9c81ee2baf2fba P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:45.136778 582 raft_consensus.cc:1261] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:45.137583 1753 consensus_queue.cc:1035] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:45.137857 1753 consensus_queue.cc:1035] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.76 ms
time per row: 0.0523802 ms
I20240503 13:58:45.179808 438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34534:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:45.184898 562 tablet_service.cc:1467] Processing CreateTablet for tablet 8d06ae5dbfb44987b060c8740bfb231e (DEFAULT_TABLE table=rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.184993 824 tablet_service.cc:1467] Processing CreateTablet for tablet 8d06ae5dbfb44987b060c8740bfb231e (DEFAULT_TABLE table=rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.185228 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d06ae5dbfb44987b060c8740bfb231e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.185315 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d06ae5dbfb44987b060c8740bfb231e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.185873 561 tablet_service.cc:1467] Processing CreateTablet for tablet 1d63d06c06ed4743b3e1176413142bef (DEFAULT_TABLE table=rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.186077 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d63d06c06ed4743b3e1176413142bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.186084 956 tablet_service.cc:1467] Processing CreateTablet for tablet 1d63d06c06ed4743b3e1176413142bef (DEFAULT_TABLE table=rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.186342 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d63d06c06ed4743b3e1176413142bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.187744 954 tablet_service.cc:1467] Processing CreateTablet for tablet b0ecd805f87a4afeb6e41b1f6f671622 (DEFAULT_TABLE table=rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.187960 954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ecd805f87a4afeb6e41b1f6f671622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.188074 693 tablet_service.cc:1467] Processing CreateTablet for tablet 8d06ae5dbfb44987b060c8740bfb231e (DEFAULT_TABLE table=rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.188344 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d06ae5dbfb44987b060c8740bfb231e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.188429 1816 tablet_bootstrap.cc:492] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:45.189532 1816 tablet_bootstrap.cc:654] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.190166 1818 tablet_bootstrap.cc:492] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576: Bootstrap starting.
I20240503 13:58:45.190244 1815 tablet_bootstrap.cc:492] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4: Bootstrap starting.
I20240503 13:58:45.190327 694 tablet_service.cc:1467] Processing CreateTablet for tablet b0ecd805f87a4afeb6e41b1f6f671622 (DEFAULT_TABLE table=rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.190536 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ecd805f87a4afeb6e41b1f6f671622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.191020 1815 tablet_bootstrap.cc:654] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.191162 1818 tablet_bootstrap.cc:654] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.192040 1816 tablet_bootstrap.cc:492] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:45.192268 1818 tablet_bootstrap.cc:492] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576: No bootstrap required, opened a new log
I20240503 13:58:45.192351 1818 ts_tablet_manager.cc:1367] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:45.192488 824 tablet_service.cc:1467] Processing CreateTablet for tablet 1d63d06c06ed4743b3e1176413142bef (DEFAULT_TABLE table=rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.192703 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d63d06c06ed4743b3e1176413142bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.192728 1817 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66: Bootstrap starting.
I20240503 13:58:45.192799 1818 raft_consensus.cc:348] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.192909 1818 raft_consensus.cc:374] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.192945 1818 raft_consensus.cc:729] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.193065 1818 consensus_queue.cc:260] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.193296 1815 tablet_bootstrap.cc:492] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4: No bootstrap required, opened a new log
I20240503 13:58:45.193360 1818 ts_tablet_manager.cc:1398] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:45.193378 1815 ts_tablet_manager.cc:1367] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:58:45.193521 1818 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: Bootstrap starting.
I20240503 13:58:45.193509 823 tablet_service.cc:1467] Processing CreateTablet for tablet b0ecd805f87a4afeb6e41b1f6f671622 (DEFAULT_TABLE table=rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.193694 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ecd805f87a4afeb6e41b1f6f671622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.193763 1815 raft_consensus.cc:348] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.193886 1815 raft_consensus.cc:374] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.193928 1815 raft_consensus.cc:729] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.194039 1815 consensus_queue.cc:260] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.194303 1815 ts_tablet_manager.cc:1398] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:45.194363 1818 tablet_bootstrap.cc:654] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.194424 1817 tablet_bootstrap.cc:654] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.195142 1816 ts_tablet_manager.cc:1367] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:58:45.195670 1816 raft_consensus.cc:348] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.195922 1815 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4: Bootstrap starting.
I20240503 13:58:45.196703 1815 tablet_bootstrap.cc:654] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.196781 1816 raft_consensus.cc:374] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.196847 1816 raft_consensus.cc:729] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.197002 1816 consensus_queue.cc:260] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.197404 1816 ts_tablet_manager.cc:1398] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:45.197814 1817 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66: No bootstrap required, opened a new log
I20240503 13:58:45.197908 1817 ts_tablet_manager.cc:1367] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:58:45.198091 1816 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:45.198375 1817 raft_consensus.cc:348] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.198482 1817 raft_consensus.cc:374] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.198516 1817 raft_consensus.cc:729] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.198684 1817 consensus_queue.cc:260] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.198272 1818 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: No bootstrap required, opened a new log
I20240503 13:58:45.199074 1818 ts_tablet_manager.cc:1367] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:58:45.199580 1818 raft_consensus.cc:348] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.199671 1818 raft_consensus.cc:374] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.199697 1818 raft_consensus.cc:729] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.199829 1818 consensus_queue.cc:260] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.199921 1816 tablet_bootstrap.cc:654] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.198988 1817 ts_tablet_manager.cc:1398] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:45.200275 1817 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: Bootstrap starting.
I20240503 13:58:45.201104 1817 tablet_bootstrap.cc:654] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.201530 1818 ts_tablet_manager.cc:1398] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:45.201546 1816 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:45.201612 1816 ts_tablet_manager.cc:1367] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:58:45.202044 1816 raft_consensus.cc:348] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.202159 1816 raft_consensus.cc:374] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.202189 1816 raft_consensus.cc:729] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.202297 1816 consensus_queue.cc:260] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.202553 1816 ts_tablet_manager.cc:1398] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:58:45.202685 1817 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: No bootstrap required, opened a new log
I20240503 13:58:45.202772 1817 ts_tablet_manager.cc:1367] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:58:45.203523 1815 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4: No bootstrap required, opened a new log
I20240503 13:58:45.203624 1815 ts_tablet_manager.cc:1367] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:58:45.203851 1817 raft_consensus.cc:348] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.203943 1817 raft_consensus.cc:374] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.203975 1817 raft_consensus.cc:729] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.204044 1815 raft_consensus.cc:348] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.204105 1817 consensus_queue.cc:260] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.204162 1815 raft_consensus.cc:374] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.204198 1815 raft_consensus.cc:729] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.204329 1815 consensus_queue.cc:260] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.204378 1817 ts_tablet_manager.cc:1398] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:45.204586 1815 ts_tablet_manager.cc:1398] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:45.204766 1815 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4: Bootstrap starting.
I20240503 13:58:45.205612 1815 tablet_bootstrap.cc:654] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.206646 1815 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4: No bootstrap required, opened a new log
I20240503 13:58:45.206733 1815 ts_tablet_manager.cc:1367] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:58:45.207100 1815 raft_consensus.cc:348] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.207172 1815 raft_consensus.cc:374] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.207204 1815 raft_consensus.cc:729] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.207351 1815 consensus_queue.cc:260] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.207571 1815 ts_tablet_manager.cc:1398] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:45.217182 1805 raft_consensus.cc:482] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:45.217291 1805 raft_consensus.cc:504] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.217738 1805 leader_election.cc:288] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:45.218077 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" is_pre_election: true
I20240503 13:58:45.218209 713 raft_consensus.cc:2454] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 0.
I20240503 13:58:45.218677 912 leader_election.cc:302] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:45.218869 1805 raft_consensus.cc:2781] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:45.218937 1805 raft_consensus.cc:482] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:45.219000 1805 raft_consensus.cc:3037] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.220548 1805 raft_consensus.cc:504] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.220921 1805 leader_election.cc:288] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Requested vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:45.221575 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:45.221699 844 raft_consensus.cc:3037] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.221709 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
I20240503 13:58:45.221823 713 raft_consensus.cc:3037] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.222440 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:45.274462 1752 raft_consensus.cc:482] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:45.274616 1752 raft_consensus.cc:504] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.274641 1714 raft_consensus.cc:482] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:45.274757 1714 raft_consensus.cc:504] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.275115 1752 leader_election.cc:288] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:45.275329 1714 leader_election.cc:288] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:45.275666 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d63d06c06ed4743b3e1176413142bef" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66" is_pre_election: true
I20240503 13:58:45.275694 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d63d06c06ed4743b3e1176413142bef" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:45.275808 845 raft_consensus.cc:2454] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 0.
I20240503 13:58:45.275810 976 raft_consensus.cc:2454] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 0.
I20240503 13:58:45.275851 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d06ae5dbfb44987b060c8740bfb231e" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:45.275970 843 raft_consensus.cc:2454] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 0.
I20240503 13:58:45.276126 518 leader_election.cc:302] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:45.276425 649 leader_election.cc:302] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:45.276460 1752 raft_consensus.cc:2781] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:45.276515 1752 raft_consensus.cc:482] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:45.276551 1752 raft_consensus.cc:3037] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.276595 1714 raft_consensus.cc:2781] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:45.276644 1714 raft_consensus.cc:482] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:45.276680 1714 raft_consensus.cc:3037] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.279342 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d06ae5dbfb44987b060c8740bfb231e" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc" is_pre_election: true
I20240503 13:58:45.279446 583 raft_consensus.cc:2454] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 0.
I20240503 13:58:45.312696 844 raft_consensus.cc:2454] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:45.312696 713 raft_consensus.cc:2454] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:45.313549 912 leader_election.cc:302] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:45.313930 1752 raft_consensus.cc:504] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.314157 1714 raft_consensus.cc:504] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.314412 1752 leader_election.cc:288] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 election: Requested vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:45.314638 1714 leader_election.cc:288] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 election: Requested vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:45.315338 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d63d06c06ed4743b3e1176413142bef" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:45.315454 844 raft_consensus.cc:3037] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.315421 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d06ae5dbfb44987b060c8740bfb231e" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:45.315541 843 raft_consensus.cc:3037] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.316341 1805 raft_consensus.cc:2781] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:45.316457 1805 raft_consensus.cc:686] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Becoming Leader. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Running, Role: LEADER
I20240503 13:58:45.316614 1805 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.316591 844 raft_consensus.cc:2454] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 1.
I20240503 13:58:45.317133 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d06ae5dbfb44987b060c8740bfb231e" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc"
I20240503 13:58:45.317175 518 leader_election.cc:302] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:45.317260 583 raft_consensus.cc:3037] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.317422 1752 raft_consensus.cc:2781] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:45.317504 1752 raft_consensus.cc:686] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 1 LEADER]: Becoming Leader. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Running, Role: LEADER
I20240503 13:58:45.317647 1752 consensus_queue.cc:237] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:45.316737 843 raft_consensus.cc:2454] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 1.
I20240503 13:58:45.318533 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d63d06c06ed4743b3e1176413142bef" candidate_uuid: "769439c9b5d3418eb0ebd0287954e6cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
I20240503 13:58:45.318583 649 leader_election.cc:302] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:45.318639 976 raft_consensus.cc:3037] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.318783 1714 raft_consensus.cc:2781] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:45.318984 1714 raft_consensus.cc:686] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Becoming Leader. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Running, Role: LEADER
I20240503 13:58:45.319133 1714 consensus_queue.cc:237] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.320300 440 catalog_manager.cc:5484] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129). New cstate: current_term: 1 leader_uuid: "769439c9b5d3418eb0ebd0287954e6cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:45.320731 438 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:45.320801 433 catalog_manager.cc:5484] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:45.389428 976 raft_consensus.cc:2454] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769439c9b5d3418eb0ebd0287954e6cc in term 1.
I20240503 13:58:45.389477 583 raft_consensus.cc:2454] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 1.
W20240503 13:58:45.465440 1836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:45.465720 1836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:45.468963 1836 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 13:58:45.552489 583 raft_consensus.cc:1261] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:45.554158 1833 consensus_queue.cc:1035] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:45.555107 843 raft_consensus.cc:1261] T 8d06ae5dbfb44987b060c8740bfb231e P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:45.555451 844 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:45.556049 976 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer 769439c9b5d3418eb0ebd0287954e6cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:45.556262 1805 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:45.556494 1752 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:45.557554 844 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 769439c9b5d3418eb0ebd0287954e6cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:45.557969 1752 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:45.559552 1833 consensus_queue.cc:1035] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:45.563280 713 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:45.563925 1805 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.659 ms
time per row: 0.0528294 ms
I20240503 13:58:45.629336 437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34534:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:58:45.635193 561 tablet_service.cc:1467] Processing CreateTablet for tablet 6e521d523c904d659b745874eec536e4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d28850114dae4436ae20500518f1e627]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.635378 694 tablet_service.cc:1467] Processing CreateTablet for tablet 6e521d523c904d659b745874eec536e4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d28850114dae4436ae20500518f1e627]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.635566 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e521d523c904d659b745874eec536e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.635663 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e521d523c904d659b745874eec536e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.635571 562 tablet_service.cc:1467] Processing CreateTablet for tablet a67ac7eb49584e3e9a09b3b3f00785b1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d28850114dae4436ae20500518f1e627]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.635977 562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67ac7eb49584e3e9a09b3b3f00785b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.636370 693 tablet_service.cc:1467] Processing CreateTablet for tablet a67ac7eb49584e3e9a09b3b3f00785b1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d28850114dae4436ae20500518f1e627]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.636572 693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67ac7eb49584e3e9a09b3b3f00785b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.637800 1818 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: Bootstrap starting.
I20240503 13:58:45.637802 1816 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:45.638163 954 tablet_service.cc:1467] Processing CreateTablet for tablet a67ac7eb49584e3e9a09b3b3f00785b1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d28850114dae4436ae20500518f1e627]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.638409 954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67ac7eb49584e3e9a09b3b3f00785b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.638653 1816 tablet_bootstrap.cc:654] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.638671 1818 tablet_bootstrap.cc:654] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.639649 823 tablet_service.cc:1467] Processing CreateTablet for tablet d8df290dcc75450b89cf9d3989ed7569 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d28850114dae4436ae20500518f1e627]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.639910 823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8df290dcc75450b89cf9d3989ed7569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.640600 1817 tablet_bootstrap.cc:492] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66: Bootstrap starting.
I20240503 13:58:45.640697 956 tablet_service.cc:1467] Processing CreateTablet for tablet d8df290dcc75450b89cf9d3989ed7569 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d28850114dae4436ae20500518f1e627]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.640918 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8df290dcc75450b89cf9d3989ed7569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.641300 1817 tablet_bootstrap.cc:654] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.641899 1815 tablet_bootstrap.cc:492] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4: Bootstrap starting.
I20240503 13:58:45.642444 1818 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: No bootstrap required, opened a new log
I20240503 13:58:45.642539 1818 ts_tablet_manager.cc:1367] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 13:58:45.642731 824 tablet_service.cc:1467] Processing CreateTablet for tablet 6e521d523c904d659b745874eec536e4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d28850114dae4436ae20500518f1e627]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.642782 1815 tablet_bootstrap.cc:654] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.642904 824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e521d523c904d659b745874eec536e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.643038 1818 raft_consensus.cc:348] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.643182 1818 raft_consensus.cc:374] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.643226 1818 raft_consensus.cc:729] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.643404 1818 consensus_queue.cc:260] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.643692 1818 ts_tablet_manager.cc:1398] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:45.643797 1817 tablet_bootstrap.cc:492] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66: No bootstrap required, opened a new log
I20240503 13:58:45.643851 1818 tablet_bootstrap.cc:492] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576: Bootstrap starting.
I20240503 13:58:45.643867 1817 ts_tablet_manager.cc:1367] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:58:45.643869 1815 tablet_bootstrap.cc:492] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4: No bootstrap required, opened a new log
I20240503 13:58:45.643946 1815 ts_tablet_manager.cc:1367] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:45.644285 1817 raft_consensus.cc:348] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.644398 1817 raft_consensus.cc:374] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.644356 1815 raft_consensus.cc:348] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.644443 1817 raft_consensus.cc:729] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.644476 1815 raft_consensus.cc:374] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.644515 1815 raft_consensus.cc:729] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.644556 1817 consensus_queue.cc:260] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.644624 1815 consensus_queue.cc:260] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.644788 1818 tablet_bootstrap.cc:654] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.644835 1817 ts_tablet_manager.cc:1398] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:58:45.644884 1815 ts_tablet_manager.cc:1398] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:58:45.645015 1815 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: Bootstrap starting.
I20240503 13:58:45.645015 1817 tablet_bootstrap.cc:492] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66: Bootstrap starting.
I20240503 13:58:45.645745 561 tablet_service.cc:1467] Processing CreateTablet for tablet d8df290dcc75450b89cf9d3989ed7569 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d28850114dae4436ae20500518f1e627]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:58:45.645828 1815 tablet_bootstrap.cc:654] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.646001 561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8df290dcc75450b89cf9d3989ed7569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:45.646016 1817 tablet_bootstrap.cc:654] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.648743 1817 tablet_bootstrap.cc:492] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66: No bootstrap required, opened a new log
I20240503 13:58:45.648839 1817 ts_tablet_manager.cc:1367] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:58:45.649296 1817 raft_consensus.cc:348] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.649412 1817 raft_consensus.cc:374] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.649447 1817 raft_consensus.cc:729] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.649591 1817 consensus_queue.cc:260] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.649655 1818 tablet_bootstrap.cc:492] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576: No bootstrap required, opened a new log
I20240503 13:58:45.649734 1818 ts_tablet_manager.cc:1367] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:58:45.650199 1818 raft_consensus.cc:348] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.650295 1818 raft_consensus.cc:374] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.650328 1818 raft_consensus.cc:729] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.650446 1818 consensus_queue.cc:260] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.650487 1815 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: No bootstrap required, opened a new log
I20240503 13:58:45.650570 1815 ts_tablet_manager.cc:1367] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 13:58:45.650710 1818 ts_tablet_manager.cc:1398] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:45.650871 1817 ts_tablet_manager.cc:1398] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:58:45.650986 1815 raft_consensus.cc:348] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.651080 1815 raft_consensus.cc:374] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.651101 1815 raft_consensus.cc:729] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.651223 1815 consensus_queue.cc:260] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.651525 1815 ts_tablet_manager.cc:1398] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:45.652891 1816 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:45.652978 1816 ts_tablet_manager.cc:1367] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240503 13:58:45.654084 1816 raft_consensus.cc:348] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.654184 1816 raft_consensus.cc:374] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.654228 1816 raft_consensus.cc:729] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.654352 1816 consensus_queue.cc:260] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.654649 1816 ts_tablet_manager.cc:1398] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 13:58:45.654848 1816 tablet_bootstrap.cc:492] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:45.655844 1816 tablet_bootstrap.cc:654] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.657478 1816 tablet_bootstrap.cc:492] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:45.657562 1816 ts_tablet_manager.cc:1367] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:58:45.657991 1816 raft_consensus.cc:348] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.658072 1816 raft_consensus.cc:374] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.658102 1816 raft_consensus.cc:729] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.658237 1816 consensus_queue.cc:260] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.658596 1816 ts_tablet_manager.cc:1398] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:58:45.658746 1816 tablet_bootstrap.cc:492] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:58:45.659574 1816 tablet_bootstrap.cc:654] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc: Neither blocks nor log segments found. Creating new log.
I20240503 13:58:45.660923 1816 tablet_bootstrap.cc:492] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc: No bootstrap required, opened a new log
I20240503 13:58:45.661013 1816 ts_tablet_manager.cc:1367] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:45.661461 1816 raft_consensus.cc:348] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.661545 1816 raft_consensus.cc:374] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:58:45.661579 1816 raft_consensus.cc:729] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:58:45.661700 1816 consensus_queue.cc:260] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.661942 1816 ts_tablet_manager.cc:1398] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:58:45.668978 1753 raft_consensus.cc:482] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:45.669111 1753 raft_consensus.cc:504] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.669561 1753 leader_election.cc:288] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:58:45.669907 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67ac7eb49584e3e9a09b3b3f00785b1" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc" is_pre_election: true
I20240503 13:58:45.670063 583 raft_consensus.cc:2454] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 0.
I20240503 13:58:45.670094 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67ac7eb49584e3e9a09b3b3f00785b1" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" is_pre_election: true
I20240503 13:58:45.670260 714 raft_consensus.cc:2454] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 0.
I20240503 13:58:45.670539 909 leader_election.cc:302] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:45.670876 1753 raft_consensus.cc:2781] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:45.670941 1753 raft_consensus.cc:482] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:45.671010 1753 raft_consensus.cc:3037] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.712334 1805 raft_consensus.cc:482] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:45.712469 1805 raft_consensus.cc:504] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.712836 1805 leader_election.cc:288] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:45.713174 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8df290dcc75450b89cf9d3989ed7569" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc" is_pre_election: true
I20240503 13:58:45.713219 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8df290dcc75450b89cf9d3989ed7569" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:45.713331 583 raft_consensus.cc:2454] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 0.
I20240503 13:58:45.713351 843 raft_consensus.cc:2454] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 0.
I20240503 13:58:45.713622 909 leader_election.cc:302] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:45.713776 1805 raft_consensus.cc:2781] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:45.713835 1805 raft_consensus.cc:482] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:45.713884 1805 raft_consensus.cc:3037] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.744542 1714 raft_consensus.cc:482] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:58:45.744735 1714 raft_consensus.cc:504] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.745221 1714 leader_election.cc:288] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:45.745652 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e521d523c904d659b745874eec536e4" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:45.745653 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e521d523c904d659b745874eec536e4" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc" is_pre_election: true
I20240503 13:58:45.745800 843 raft_consensus.cc:2454] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 0.
I20240503 13:58:45.745831 583 raft_consensus.cc:2454] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 0.
I20240503 13:58:45.746076 649 leader_election.cc:302] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:45.746256 1714 raft_consensus.cc:2781] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:58:45.746311 1714 raft_consensus.cc:482] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:58:45.746337 1714 raft_consensus.cc:3037] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.769549 1753 raft_consensus.cc:504] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.769549 1714 raft_consensus.cc:504] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.770041 1753 leader_election.cc:288] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Requested vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:58:45.770026 1714 leader_election.cc:288] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 election: Requested vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:45.770864 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67ac7eb49584e3e9a09b3b3f00785b1" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
I20240503 13:58:45.770839 583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e521d523c904d659b745874eec536e4" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc"
I20240503 13:58:45.770988 714 raft_consensus.cc:3037] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.770993 583 raft_consensus.cc:3037] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.771544 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67ac7eb49584e3e9a09b3b3f00785b1" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc"
I20240503 13:58:45.771672 582 raft_consensus.cc:3037] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.772009 714 raft_consensus.cc:2454] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:45.772224 583 raft_consensus.cc:2454] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 1.
I20240503 13:58:45.772315 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e521d523c904d659b745874eec536e4" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:45.772447 843 raft_consensus.cc:3037] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.772643 582 raft_consensus.cc:2454] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:45.772866 912 leader_election.cc:302] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:45.773046 1753 raft_consensus.cc:2781] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:45.773118 1753 raft_consensus.cc:686] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Becoming Leader. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Running, Role: LEADER
I20240503 13:58:45.771168 1805 raft_consensus.cc:504] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.773272 1753 consensus_queue.cc:237] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:45.773450 843 raft_consensus.cc:2454] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 1.
I20240503 13:58:45.773641 1805 leader_election.cc:288] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Requested vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:45.773669 647 leader_election.cc:302] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, 769439c9b5d3418eb0ebd0287954e6cc; no voters:
I20240503 13:58:45.774081 1714 raft_consensus.cc:2781] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:45.774170 1714 raft_consensus.cc:686] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Becoming Leader. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Running, Role: LEADER
I20240503 13:58:45.774298 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8df290dcc75450b89cf9d3989ed7569" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc"
I20240503 13:58:45.774293 1714 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.774389 582 raft_consensus.cc:3037] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.775208 582 raft_consensus.cc:2454] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:45.775332 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8df290dcc75450b89cf9d3989ed7569" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:45.775457 843 raft_consensus.cc:3037] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:58:45.775722 909 leader_election.cc:302] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769439c9b5d3418eb0ebd0287954e6cc, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:58:45.775884 1803 raft_consensus.cc:2781] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:58:45.775954 1803 raft_consensus.cc:686] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Becoming Leader. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Running, Role: LEADER
I20240503 13:58:45.776057 1803 consensus_queue.cc:237] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:45.776244 437 catalog_manager.cc:5484] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:45.776518 843 raft_consensus.cc:2454] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:45.778137 441 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:45.780012 437 catalog_manager.cc:5484] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:45.916337 1889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:45.916586 1889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:45.919646 1889 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 13:58:46.006739 843 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:46.007926 1833 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:46.008953 582 raft_consensus.cc:1261] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:46.009387 1803 consensus_queue.cc:1035] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:46.013267 843 raft_consensus.cc:1261] T d8df290dcc75450b89cf9d3989ed7569 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:46.013657 1805 consensus_queue.cc:1035] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:46.022450 714 raft_consensus.cc:1261] T a67ac7eb49584e3e9a09b3b3f00785b1 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:46.022979 1753 consensus_queue.cc:1035] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:46.023375 582 raft_consensus.cc:1261] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:46.027768 1753 consensus_queue.cc:1035] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:46.032256 582 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:58:46.032657 1833 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:58:46.044281 1809 mvcc.cc:205] Tried to move back new op lower bound from 7023594397734219776 to 7023594396768624640. Current Snapshot: MvccSnapshot[applied={T|T < 7023594397734219776}]
INSERT report
rows total: 2000
time total: 115.442 ms
time per row: 0.0577208 ms
I20240503 13:58:46.084800 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:34511
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:40943
--webserver_port=39727
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 13:58:46.199060 1922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:46.199319 1922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:46.199410 1922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:46.202371 1922 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:46.202430 1922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:46.202535 1922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:58:46.205932 1922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:34511
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=39727
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.1922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:46.206777 1922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:46.207906 1922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:46.216943 1930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:46.217767 1928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:46.218593 1927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:46.219995 1922 server_base.cc:1034] running on GCE node
I20240503 13:58:46.220398 1922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:46.220970 1922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:46.222129 1922 hybrid_clock.cc:648] HybridClock initialized: now 1714744726222110 us; error 41 us; skew 500 ppm
I20240503 13:58:46.224057 1922 webserver.cc:460] Webserver started at http://127.0.99.133:39727/ using document root <none> and password file <none>
I20240503 13:58:46.224617 1922 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:46.224699 1922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:46.227854 1922 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 13:58:46.229722 1936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.230696 1922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 13:58:46.230782 1922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "83070528d9574a528c6e57fd30b33dcc"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:46.231176 1922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:46.241014 1922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:46.241544 1922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:46.241695 1922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:46.242175 1922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:46.243005 1922 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:46.243055 1922 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.243111 1922 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:46.243145 1922 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.251562 1922 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34511
I20240503 13:58:46.251578 2049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34511 every 8 connection(s)
I20240503 13:58:46.252640 1922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:58:46.260690 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 1922
I20240503 13:58:46.261118 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:36679
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:40943
--webserver_port=33045
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 13:58:46.261615 2050 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:46.261838 2050 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:46.262439 2050 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:46.263770 441 ts_manager.cc:194] Re-registered known tserver with Master: 83070528d9574a528c6e57fd30b33dcc (127.0.99.133:34511)
I20240503 13:58:46.264565 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:34685
W20240503 13:58:46.376296 2053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:46.376506 2053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:46.376554 2053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:46.379581 2053 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:46.379647 2053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:46.379746 2053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:58:46.383203 2053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:36679
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=33045
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.2053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:46.384068 2053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:46.385169 2053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:46.394476 2059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:46.395838 2061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:46.397009 2058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:46.398157 2053 server_base.cc:1034] running on GCE node
I20240503 13:58:46.398819 2053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:46.399932 2053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:46.401258 2053 hybrid_clock.cc:648] HybridClock initialized: now 1714744726401226 us; error 66 us; skew 500 ppm
I20240503 13:58:46.403757 2053 webserver.cc:460] Webserver started at http://127.0.99.134:33045/ using document root <none> and password file <none>
I20240503 13:58:46.404278 2053 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:46.404341 2053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:46.407568 2053 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:58:46.409421 2067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.410431 2053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:46.410531 2053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3be34dcec5ad4ec7860fe6145ac58163"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:58:46.410974 2053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:46.428478 2053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:46.429088 2053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:46.429231 2053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:46.429728 2053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:46.430549 2053 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:46.430593 2053 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.430632 2053 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:46.430653 2053 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.439936 2053 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:36679
I20240503 13:58:46.440025 2180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:36679 every 8 connection(s)
I20240503 13:58:46.441058 2053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:58:46.447949 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 2053
I20240503 13:58:46.448436 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:41905
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:40943
--webserver_port=41317
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 13:58:46.449877 2181 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:46.450150 2181 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:46.450695 2181 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:46.451747 441 ts_manager.cc:194] Re-registered known tserver with Master: 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134:36679)
I20240503 13:58:46.452558 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:56197
W20240503 13:58:46.570678 2184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:46.570914 2184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:46.570962 2184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:46.574110 2184 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:46.574182 2184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:46.574290 2184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:58:46.578269 2184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:41905
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=41317
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.2184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:46.579324 2184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:46.580921 2184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:46.590979 2189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:46.591377 2192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:46.592252 2190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:46.593207 2184 server_base.cc:1034] running on GCE node
I20240503 13:58:46.593628 2184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:46.594379 2184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:46.595536 2184 hybrid_clock.cc:648] HybridClock initialized: now 1714744726595519 us; error 51 us; skew 500 ppm
I20240503 13:58:46.597322 2184 webserver.cc:460] Webserver started at http://127.0.99.135:41317/ using document root <none> and password file <none>
I20240503 13:58:46.597885 2184 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:46.597960 2184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:46.601307 2184 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:58:46.603153 2198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.604187 2184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:58:46.604282 2184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ab254774096345abb0c1daf0f7afecac"
format_stamp: "Formatted at 2024-05-03 13:58:39 on dist-test-slave-snql"
I20240503 13:58:46.604707 2184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:46.624755 2184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:46.625365 2184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:46.625521 2184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:46.626056 2184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:46.626920 2184 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:46.626971 2184 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.627033 2184 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:46.627058 2184 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.636106 2184 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:41905
I20240503 13:58:46.636139 2311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:41905 every 8 connection(s)
I20240503 13:58:46.637120 2184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:58:46.645326 2312 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:46.645392 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 2184
I20240503 13:58:46.645576 2312 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:46.645807 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:33227
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:40943
--webserver_port=37193
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 13:58:46.646194 2312 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:46.647316 441 ts_manager.cc:194] Re-registered known tserver with Master: ab254774096345abb0c1daf0f7afecac (127.0.99.135:41905)
I20240503 13:58:46.648079 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:39039
W20240503 13:58:46.770283 2315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:46.770553 2315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:58:46.770645 2315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:46.773861 2315 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:58:46.773950 2315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:58:46.774055 2315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 13:58:46.777825 2315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:33227
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=37193
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.2315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:58:46.778832 2315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:58:46.780030 2315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:58:46.789364 2321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:46.790320 2320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:58:46.791162 2323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:58:46.792017 2315 server_base.cc:1034] running on GCE node
I20240503 13:58:46.792399 2315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:58:46.793052 2315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:58:46.794210 2315 hybrid_clock.cc:648] HybridClock initialized: now 1714744726794184 us; error 39 us; skew 500 ppm
I20240503 13:58:46.796149 2315 webserver.cc:460] Webserver started at http://127.0.99.136:37193/ using document root <none> and password file <none>
I20240503 13:58:46.796702 2315 fs_manager.cc:359] Metadata directory not provided
I20240503 13:58:46.796777 2315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:58:46.800232 2315 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 13:58:46.802145 2329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.803215 2315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 13:58:46.803351 2315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "06de56aefa784b4c8b7d5a3e7c5f9825"
format_stamp: "Formatted at 2024-05-03 13:58:39 on dist-test-slave-snql"
I20240503 13:58:46.803781 2315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:58:46.820924 2315 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:58:46.821542 2315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:58:46.821702 2315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:58:46.822253 2315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:58:46.823202 2315 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:58:46.823288 2315 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.823352 2315 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:58:46.823379 2315 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:58:46.833727 2315 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:33227
I20240503 13:58:46.833750 2442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:33227 every 8 connection(s)
I20240503 13:58:46.834859 2315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:58:46.843585 2443 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:58:46.843621 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 2315
I20240503 13:58:46.843849 2443 heartbeater.cc:464] Registering TS with master...
I20240503 13:58:46.844480 2443 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:58:46.845618 441 ts_manager.cc:194] Re-registered known tserver with Master: 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136:33227)
I20240503 13:58:46.846433 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:36371
I20240503 13:58:47.055934 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.056771 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:47.057494 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:58:47.057760 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.079056 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:47.086568 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.088672 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.089542 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:47.262936 437 ts_manager.cc:292] Set tserver state for 769439c9b5d3418eb0ebd0287954e6cc to MAINTENANCE_MODE
I20240503 13:58:47.266583 2050 heartbeater.cc:503] Master 127.0.99.190:40943 was elected leader, sending a full tablet report...
W20240503 13:58:47.382280 2507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:47.382591 2507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:47.385789 2507 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:58:47.454478 2181 heartbeater.cc:503] Master 127.0.99.190:40943 was elected leader, sending a full tablet report...
I20240503 13:58:47.484947 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.486569 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.497975 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:58:47.498591 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.501657 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.505271 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:47.513770 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.520080 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:58:47.523219 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:47.525159 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.527411 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:47.532193 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.566993 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.573369 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:47.573984 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:47.582891 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:47.650213 2312 heartbeater.cc:503] Master 127.0.99.190:40943 was elected leader, sending a full tablet report...
I20240503 13:58:47.693039 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.696748 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.702980 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:58:47.703877 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.716051 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:47.716353 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:47.717700 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.734336 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 13:58:47.805014 2602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:47.805279 2602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:47.808555 2602 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:58:47.851492 2443 heartbeater.cc:503] Master 127.0.99.190:40943 was elected leader, sending a full tablet report...
I20240503 13:58:47.854493 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.855584 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.856513 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.864204 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:47.877315 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.879712 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:47.891381 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:58:47.900166 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:47.937520 582 consensus_queue.cc:237] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:47.938935 976 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer 769439c9b5d3418eb0ebd0287954e6cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:58:47.939785 843 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 769439c9b5d3418eb0ebd0287954e6cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:58:47.940790 2699 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:58:47.940999 1575 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:58:47.943564 1575 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } }
I20240503 13:58:47.946143 843 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } }
I20240503 13:58:47.947049 438 catalog_manager.cc:5484] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "769439c9b5d3418eb0ebd0287954e6cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:47.949445 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.949976 976 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } }
I20240503 13:58:47.955732 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.956816 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:58:47.965632 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.966423 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:47.966903 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:47.968405 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:47.985280 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:47.993482 582 consensus_queue.cc:237] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:58:47.995069 843 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 769439c9b5d3418eb0ebd0287954e6cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:47.995508 1575 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:47.996858 976 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer 769439c9b5d3418eb0ebd0287954e6cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:47.997180 2699 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:58:47.997639 843 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:58:47.998956 425 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1d63d06c06ed4743b3e1176413142bef with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 13:58:47.999428 1575 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:47.999898 434 catalog_manager.cc:5484] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc reported cstate change: config changed from index 4 to 5, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "769439c9b5d3418eb0ebd0287954e6cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:48.000450 976 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
W20240503 13:58:48.006908 517 consensus_peers.cc:485] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc -> Peer ab254774096345abb0c1daf0f7afecac (127.0.99.135:41905): Couldn't send request to peer ab254774096345abb0c1daf0f7afecac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d63d06c06ed4743b3e1176413142bef. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:48.013159 582 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d63d06c06ed4743b3e1176413142bef"
dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38472
I20240503 13:58:48.013329 582 raft_consensus.cc:595] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 1 LEADER]: Received request to transfer leadership
W20240503 13:58:48.154371 2728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:48.154582 2728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:48.157367 2728 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:58:48.232972 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:48.235774 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:48.237726 956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:58:48.239748 562 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:58:48.240753 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:48.245083 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:48.266573 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:48.270720 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 13:58:48.422317 2771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:48.422528 2771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:48.425402 2771 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:58:48.432035 1575 raft_consensus.cc:979] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc: : Instructing follower e59e5d411bc046a28486330e3e4ffb66 to start an election
I20240503 13:58:48.432152 1575 raft_consensus.cc:1067] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 1 LEADER]: Signalling peer e59e5d411bc046a28486330e3e4ffb66 to start an election
I20240503 13:58:48.432719 976 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d63d06c06ed4743b3e1176413142bef"
dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
from {username='slave'} at 127.0.99.129:37471
I20240503 13:58:48.432910 976 raft_consensus.cc:482] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:58:48.432968 976 raft_consensus.cc:3037] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:48.434157 976 raft_consensus.cc:504] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:58:48.434605 976 leader_election.cc:288] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 election: Requested vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:58:48.435055 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d63d06c06ed4743b3e1176413142bef" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:48.435053 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d63d06c06ed4743b3e1176413142bef" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "769439c9b5d3418eb0ebd0287954e6cc"
I20240503 13:58:48.435173 843 raft_consensus.cc:3037] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:48.435173 582 raft_consensus.cc:3032] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:58:48.435216 582 raft_consensus.cc:729] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Running, Role: LEADER
I20240503 13:58:48.435400 582 consensus_queue.cc:260] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:58:48.435566 582 raft_consensus.cc:3037] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:48.436211 843 raft_consensus.cc:2454] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 2.
I20240503 13:58:48.436573 582 raft_consensus.cc:2454] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 2.
I20240503 13:58:48.436561 911 leader_election.cc:302] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e59e5d411bc046a28486330e3e4ffb66, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:48.436874 844 raft_consensus.cc:1226] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Rejecting Update request from peer 769439c9b5d3418eb0ebd0287954e6cc for earlier term 1. Current term is 2. Ops: []
I20240503 13:58:48.436872 2629 raft_consensus.cc:2781] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:48.437019 2629 raft_consensus.cc:686] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 2 LEADER]: Becoming Leader. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Running, Role: LEADER
I20240503 13:58:48.437160 2629 consensus_queue.cc:237] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:58:48.439083 438 catalog_manager.cc:5484] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: term changed from 1 to 2, leader changed from 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) to e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:48.502528 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:48.504416 562 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:48.511795 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:48.514703 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:48.515779 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:48.516908 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:48.517887 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:48.538465 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Runtime error: ksck discovered errors
W20240503 13:58:48.689414 2814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:48.689661 2814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:48.692477 2814 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:58:48.775797 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:48.778475 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:48.781157 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:48.784490 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:48.784679 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:48.784695 562 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:48.785323 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:48.786895 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240503 13:58:48.922199 582 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:48.922695 2786 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:48.926451 844 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:48.926887 2864 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 13:58:48.933076 910 consensus_peers.cc:485] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 -> Peer ab254774096345abb0c1daf0f7afecac (127.0.99.135:41905): Couldn't send request to peer ab254774096345abb0c1daf0f7afecac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d63d06c06ed4743b3e1176413142bef. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:58:48.964372 2862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:48.964603 2862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:48.967564 2862 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:58:49.045974 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.048903 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:49.051826 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.057905 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.058125 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.059274 562 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:49.087625 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:49.090313 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 13:58:49.237152 2910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:49.237390 2910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:49.240746 2910 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:58:49.320979 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.321813 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.324836 562 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:49.327198 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:49.327396 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:49.334218 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:58:49.341760 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.343214 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.405790 2954 ts_tablet_manager.cc:902] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: Initiating tablet copy from peer e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:49.406607 2954 tablet_copy_client.cc:308] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40971
I20240503 13:58:49.410935 996 tablet_copy_service.cc:140] P e59e5d411bc046a28486330e3e4ffb66: Received BeginTabletCopySession request for tablet 1d63d06c06ed4743b3e1176413142bef from peer ab254774096345abb0c1daf0f7afecac ({username='slave'} at 127.0.99.135:45033)
I20240503 13:58:49.411085 996 tablet_copy_service.cc:161] P e59e5d411bc046a28486330e3e4ffb66: Beginning new tablet copy session on tablet 1d63d06c06ed4743b3e1176413142bef from peer ab254774096345abb0c1daf0f7afecac at {username='slave'} at 127.0.99.135:45033: session id = ab254774096345abb0c1daf0f7afecac-1d63d06c06ed4743b3e1176413142bef
I20240503 13:58:49.412715 996 tablet_copy_source_session.cc:205] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:49.416148 2954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d63d06c06ed4743b3e1176413142bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:49.421214 2954 tablet_copy_client.cc:783] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:49.421407 2954 tablet_copy_client.cc:647] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:49.423216 2958 tablet_copy_client.cc:1002] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: tablet copy: Injecting latency into file download: 1500
W20240503 13:58:49.513449 2953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:58:49.513670 2953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:58:49.516698 2953 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:58:49.593263 562 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:49.593464 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:58:49.594480 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.601958 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.605983 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.609594 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:58:49.610373 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:58:49.634596 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240503 13:58:49.668287 2505 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 501
W20240503 13:58:49.689839 909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:58:49.690590 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:58:49.729699 647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:58:49.730502 647 consensus_peers.cc:485] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:58:49.803165 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:58:49.857468 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:58:49.979722 778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:58:49.980599 778 consensus_peers.cc:485] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:58:50.011402 909 consensus_peers.cc:485] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:58:50.068826 909 consensus_peers.cc:485] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:58:50.193857 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:58:50.252431 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:50.741637 3032 raft_consensus.cc:482] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 769439c9b5d3418eb0ebd0287954e6cc)
I20240503 13:58:50.741772 3032 raft_consensus.cc:504] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:50.742189 3032 leader_election.cc:288] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:50.742800 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd93f79958e489b9028f5123c600f74" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66" is_pre_election: true
W20240503 13:58:50.743151 778 leader_election.cc:334] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:50.743340 778 leader_election.cc:302] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58fb372531d42a3bc405510bf5bced4; no voters: 769439c9b5d3418eb0ebd0287954e6cc, e59e5d411bc046a28486330e3e4ffb66
I20240503 13:58:50.743472 3032 raft_consensus.cc:2726] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:58:50.885234 3032 raft_consensus.cc:482] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 769439c9b5d3418eb0ebd0287954e6cc)
I20240503 13:58:50.885353 3032 raft_consensus.cc:504] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:50.885726 3032 leader_election.cc:288] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:58:50.886039 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ee1c64f81b492a9c83e7618878ff79" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" is_pre_election: true
W20240503 13:58:50.886696 778 leader_election.cc:334] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:50.886762 778 leader_election.cc:302] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58fb372531d42a3bc405510bf5bced4; no voters: 37dbd0b8eaae46d59b318fbbb4bab576, 769439c9b5d3418eb0ebd0287954e6cc
I20240503 13:58:50.886907 3032 raft_consensus.cc:2726] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:58:50.923902 2954 tablet_copy_client.cc:523] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:50.925801 2954 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: Bootstrap starting.
I20240503 13:58:50.931208 2954 log.cc:824] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: Log is configured to *not* fsync() on all Append() calls
I20240503 13:58:50.950935 2954 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:50.951332 2954 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: Bootstrap complete.
I20240503 13:58:50.951628 2954 ts_tablet_manager.cc:1367] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.004s
I20240503 13:58:50.953480 2954 raft_consensus.cc:348] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:58:50.953696 2954 raft_consensus.cc:729] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ab254774096345abb0c1daf0f7afecac, State: Initialized, Role: LEARNER
I20240503 13:58:50.953969 2954 consensus_queue.cc:260] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:58:50.954768 2954 ts_tablet_manager.cc:1398] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 13:58:50.955577 996 tablet_copy_service.cc:342] P e59e5d411bc046a28486330e3e4ffb66: Request end of tablet copy session ab254774096345abb0c1daf0f7afecac-1d63d06c06ed4743b3e1176413142bef received from {username='slave'} at 127.0.99.135:45033
I20240503 13:58:50.955698 996 tablet_copy_service.cc:433] P e59e5d411bc046a28486330e3e4ffb66: ending tablet copy session ab254774096345abb0c1daf0f7afecac-1d63d06c06ed4743b3e1176413142bef on tablet 1d63d06c06ed4743b3e1176413142bef with peer ab254774096345abb0c1daf0f7afecac
I20240503 13:58:50.972301 2266 raft_consensus.cc:1203] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Deduplicated request from leader. Original: 1.5->[2.6-2.6] Dedup: 2.6->[]
I20240503 13:58:50.975486 3025 raft_consensus.cc:482] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 769439c9b5d3418eb0ebd0287954e6cc)
I20240503 13:58:50.975611 3025 raft_consensus.cc:504] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:50.975998 3025 leader_election.cc:288] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:58:50.976375 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ee1c64f81b492a9c83e7618878ff79" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:50.976517 844 raft_consensus.cc:2454] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 1.
I20240503 13:58:50.976838 649 leader_election.cc:302] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:58:50.977013 3025 raft_consensus.cc:2781] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:58:50.977067 3025 raft_consensus.cc:482] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 769439c9b5d3418eb0ebd0287954e6cc)
I20240503 13:58:50.977115 3025 raft_consensus.cc:3037] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:58:50.977193 647 leader_election.cc:334] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:50.977974 3025 raft_consensus.cc:504] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:50.978255 3025 leader_election.cc:288] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 2 election: Requested vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:58:50.978564 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67ee1c64f81b492a9c83e7618878ff79" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:50.978677 844 raft_consensus.cc:3037] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:58:50.979118 647 leader_election.cc:334] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:50.979627 844 raft_consensus.cc:2454] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 2.
I20240503 13:58:50.979918 649 leader_election.cc:302] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, f58fb372531d42a3bc405510bf5bced4; no voters: 769439c9b5d3418eb0ebd0287954e6cc
I20240503 13:58:50.980105 3025 raft_consensus.cc:2781] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:50.980278 3025 raft_consensus.cc:686] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Becoming Leader. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Running, Role: LEADER
I20240503 13:58:50.980430 3032 raft_consensus.cc:482] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 769439c9b5d3418eb0ebd0287954e6cc)
I20240503 13:58:50.980415 3025 consensus_queue.cc:237] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:58:50.980511 3032 raft_consensus.cc:504] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:50.980857 3032 leader_election.cc:288] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:50.981453 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35914f78ffe423e91ce866785764b9c" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e59e5d411bc046a28486330e3e4ffb66" is_pre_election: true
W20240503 13:58:50.981827 778 leader_election.cc:334] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:50.981887 778 leader_election.cc:302] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58fb372531d42a3bc405510bf5bced4; no voters: 769439c9b5d3418eb0ebd0287954e6cc, e59e5d411bc046a28486330e3e4ffb66
I20240503 13:58:50.981993 3032 raft_consensus.cc:2726] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:58:50.982172 441 catalog_manager.cc:5484] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: term changed from 1 to 2, leader changed from 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) to 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:51.054585 3018 raft_consensus.cc:482] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 769439c9b5d3418eb0ebd0287954e6cc)
I20240503 13:58:51.054749 3018 raft_consensus.cc:504] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:51.055166 3018 leader_election.cc:288] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:58:51.055568 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd93f79958e489b9028f5123c600f74" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:51.055732 844 raft_consensus.cc:2454] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
W20240503 13:58:51.055761 909 leader_election.cc:334] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:51.056034 911 leader_election.cc:302] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e59e5d411bc046a28486330e3e4ffb66, f58fb372531d42a3bc405510bf5bced4; no voters: 769439c9b5d3418eb0ebd0287954e6cc
I20240503 13:58:51.056206 3018 raft_consensus.cc:2781] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:58:51.056269 3018 raft_consensus.cc:482] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 769439c9b5d3418eb0ebd0287954e6cc)
I20240503 13:58:51.056304 3018 raft_consensus.cc:3037] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:51.057168 3018 raft_consensus.cc:504] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:51.057462 3018 leader_election.cc:288] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 election: Requested vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:58:51.057830 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd93f79958e489b9028f5123c600f74" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:51.057936 844 raft_consensus.cc:3037] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:58:51.058610 909 leader_election.cc:334] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:51.059036 844 raft_consensus.cc:2454] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 2.
I20240503 13:58:51.059451 911 leader_election.cc:302] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e59e5d411bc046a28486330e3e4ffb66, f58fb372531d42a3bc405510bf5bced4; no voters: 769439c9b5d3418eb0ebd0287954e6cc
I20240503 13:58:51.059638 3018 raft_consensus.cc:2781] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:51.059731 3018 raft_consensus.cc:686] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [term 2 LEADER]: Becoming Leader. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Running, Role: LEADER
I20240503 13:58:51.059868 3018 consensus_queue.cc:237] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:58:51.061432 441 catalog_manager.cc:5484] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: term changed from 1 to 2, leader changed from 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) to e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:51.077512 3018 raft_consensus.cc:482] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 769439c9b5d3418eb0ebd0287954e6cc)
I20240503 13:58:51.077673 3018 raft_consensus.cc:504] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:51.078125 3018 leader_election.cc:288] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:51.078640 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35914f78ffe423e91ce866785764b9c" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4" is_pre_election: true
I20240503 13:58:51.078794 844 raft_consensus.cc:2454] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 1.
I20240503 13:58:51.079097 911 leader_election.cc:302] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e59e5d411bc046a28486330e3e4ffb66, f58fb372531d42a3bc405510bf5bced4; no voters:
W20240503 13:58:51.079187 909 leader_election.cc:334] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:51.079267 3018 raft_consensus.cc:2781] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:58:51.079314 3018 raft_consensus.cc:482] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 769439c9b5d3418eb0ebd0287954e6cc)
I20240503 13:58:51.079365 3018 raft_consensus.cc:3037] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:58:51.080209 3018 raft_consensus.cc:504] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:51.080513 3018 leader_election.cc:288] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 election: Requested vote from peers 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075), f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:51.080899 844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35914f78ffe423e91ce866785764b9c" candidate_uuid: "e59e5d411bc046a28486330e3e4ffb66" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:58:51.081004 844 raft_consensus.cc:3037] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:58:51.081393 909 leader_election.cc:334] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:51.081804 844 raft_consensus.cc:2454] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59e5d411bc046a28486330e3e4ffb66 in term 2.
I20240503 13:58:51.082082 911 leader_election.cc:302] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e59e5d411bc046a28486330e3e4ffb66, f58fb372531d42a3bc405510bf5bced4; no voters: 769439c9b5d3418eb0ebd0287954e6cc
I20240503 13:58:51.082219 3018 raft_consensus.cc:2781] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:58:51.082306 3018 raft_consensus.cc:686] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [term 2 LEADER]: Becoming Leader. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Running, Role: LEADER
I20240503 13:58:51.082441 3018 consensus_queue.cc:237] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:58:51.084257 441 catalog_manager.cc:5484] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: term changed from 1 to 2, leader changed from 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) to e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:51.427285 647 consensus_peers.cc:485] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.459332 3018 raft_consensus.cc:1050] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66: attempting to promote NON_VOTER ab254774096345abb0c1daf0f7afecac to VOTER
I20240503 13:58:51.460033 3018 consensus_queue.cc:237] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:58:51.461514 844 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 13:58:51.461710 2266 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 13:58:51.462112 3068 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:58:51.462275 3018 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 13:58:51.463841 909 consensus_peers.cc:485] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.468174 3068 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:51.468722 2266 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:51.470127 844 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:51.470731 441 catalog_manager.cc:5484] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index 5 to 7, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:51.473495 976 consensus_queue.cc:237] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:58:51.474655 844 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 13:58:51.474673 2266 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 13:58:51.474970 3070 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:58:51.475096 2960 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:58:51.476526 2960 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:51.476907 2266 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:51.477974 425 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d63d06c06ed4743b3e1176413142bef with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:58:51.478257 844 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:51.478683 434 catalog_manager.cc:5484] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index 7 to 8, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:58:51.481611 425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:58:51.482434 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1d63d06c06ed4743b3e1176413142bef: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:51.497082 844 raft_consensus.cc:1261] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:58:51.497545 3025 consensus_queue.cc:1035] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:58:51.515803 844 raft_consensus.cc:1261] T a35914f78ffe423e91ce866785764b9c P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:58:51.516183 2960 consensus_queue.cc:1035] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:58:51.519572 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 13:58:51.542093 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.548173 844 raft_consensus.cc:1261] T 3fd93f79958e489b9028f5123c600f74 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 13:58:51.548566 3070 consensus_queue.cc:1035] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:58:51.764335 713 consensus_queue.cc:237] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } }
I20240503 13:58:51.765573 844 raft_consensus.cc:1261] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 13:58:51.765934 647 consensus_peers.cc:485] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.766007 3025 consensus_queue.cc:1035] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:51.767426 3076 raft_consensus.cc:2932] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } }
I20240503 13:58:51.767779 844 raft_consensus.cc:2932] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } }
I20240503 13:58:51.769810 441 catalog_manager.cc:5484] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:51.772529 713 consensus_queue.cc:237] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:58:51.773880 844 raft_consensus.cc:1261] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 13:58:51.774243 3098 consensus_queue.cc:1035] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 13:58:51.775429 647 consensus_peers.cc:485] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.775926 3025 raft_consensus.cc:2932] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:58:51.776245 844 raft_consensus.cc:2932] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:58:51.778101 428 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 67ee1c64f81b492a9c83e7618878ff79 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 13:58:51.778173 648 consensus_peers.cc:485] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer ab254774096345abb0c1daf0f7afecac (127.0.99.135:41905): Couldn't send request to peer ab254774096345abb0c1daf0f7afecac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67ee1c64f81b492a9c83e7618878ff79. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:51.778148 441 catalog_manager.cc:5484] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: config changed from index 5 to 6, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:52.124890 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:58:52.169237 3107 ts_tablet_manager.cc:902] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac: Initiating tablet copy from peer 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:58:52.169689 3107 tablet_copy_client.cc:308] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42209
I20240503 13:58:52.173209 734 tablet_copy_service.cc:140] P 37dbd0b8eaae46d59b318fbbb4bab576: Received BeginTabletCopySession request for tablet 67ee1c64f81b492a9c83e7618878ff79 from peer ab254774096345abb0c1daf0f7afecac ({username='slave'} at 127.0.99.135:57933)
I20240503 13:58:52.173352 734 tablet_copy_service.cc:161] P 37dbd0b8eaae46d59b318fbbb4bab576: Beginning new tablet copy session on tablet 67ee1c64f81b492a9c83e7618878ff79 from peer ab254774096345abb0c1daf0f7afecac at {username='slave'} at 127.0.99.135:57933: session id = ab254774096345abb0c1daf0f7afecac-67ee1c64f81b492a9c83e7618878ff79
I20240503 13:58:52.174504 734 tablet_copy_source_session.cc:205] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:52.175639 3107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67ee1c64f81b492a9c83e7618878ff79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:52.177398 3107 tablet_copy_client.cc:783] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:52.177559 3107 tablet_copy_client.cc:647] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:52.178786 3111 tablet_copy_client.cc:1002] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac: tablet copy: Injecting latency into file download: 1500
W20240503 13:58:52.196071 647 consensus_peers.cc:485] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:58:52.232461 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:58:52.261581 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:58:52.267589 778 consensus_peers.cc:485] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:58:52.489909 909 consensus_peers.cc:485] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:58:52.691888 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1d63d06c06ed4743b3e1176413142bef: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:58:52.757107 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:58:52.778174 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:58:53.679118 3107 tablet_copy_client.cc:523] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:53.680462 3107 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac: Bootstrap starting.
I20240503 13:58:53.697065 3107 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:53.697322 3107 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac: Bootstrap complete.
I20240503 13:58:53.697455 3107 ts_tablet_manager.cc:1367] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20240503 13:58:53.697886 3107 raft_consensus.cc:348] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:58:53.697994 3107 raft_consensus.cc:729] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ab254774096345abb0c1daf0f7afecac, State: Initialized, Role: LEARNER
I20240503 13:58:53.698071 3107 consensus_queue.cc:260] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:58:53.698321 3107 ts_tablet_manager.cc:1398] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 13:58:53.698902 734 tablet_copy_service.cc:342] P 37dbd0b8eaae46d59b318fbbb4bab576: Request end of tablet copy session ab254774096345abb0c1daf0f7afecac-67ee1c64f81b492a9c83e7618878ff79 received from {username='slave'} at 127.0.99.135:57933
I20240503 13:58:53.699009 734 tablet_copy_service.cc:433] P 37dbd0b8eaae46d59b318fbbb4bab576: ending tablet copy session ab254774096345abb0c1daf0f7afecac-67ee1c64f81b492a9c83e7618878ff79 on tablet 67ee1c64f81b492a9c83e7618878ff79 with peer ab254774096345abb0c1daf0f7afecac
W20240503 13:58:53.760279 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1d63d06c06ed4743b3e1176413142bef: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:58:53.902668 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:58:53.964531 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:58:53.984405 647 consensus_peers.cc:485] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:58:54.262660 2266 raft_consensus.cc:1203] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240503 13:58:54.536742 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:58:54.588567 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:58:54.701996 778 consensus_peers.cc:485] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:58:54.704200 909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 13:58:54.704741 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:58:54.760277 3193 raft_consensus.cc:1050] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576: attempting to promote NON_VOTER ab254774096345abb0c1daf0f7afecac to VOTER
I20240503 13:58:54.760681 3193 consensus_queue.cc:237] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:58:54.761945 2266 raft_consensus.cc:1261] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240503 13:58:54.762006 647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 13:58:54.762126 844 raft_consensus.cc:1261] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 13:58:54.762374 3192 consensus_queue.cc:1035] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:58:54.762704 3195 consensus_queue.cc:1035] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 13:58:54.762857 647 consensus_peers.cc:485] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:54.764405 3195 raft_consensus.cc:2932] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:54.764794 2266 raft_consensus.cc:2932] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:54.766759 844 raft_consensus.cc:2932] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:54.767264 438 catalog_manager.cc:5484] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: config changed from index 6 to 7, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:58:54.770438 713 consensus_queue.cc:237] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:58:54.771543 2266 raft_consensus.cc:1261] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 13:58:54.771629 844 raft_consensus.cc:1261] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 13:58:54.771850 3193 consensus_queue.cc:1035] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:58:54.771956 3192 consensus_queue.cc:1035] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:58:54.773325 3192 raft_consensus.cc:2932] T 67ee1c64f81b492a9c83e7618878ff79 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:54.773586 2266 raft_consensus.cc:2932] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:54.773736 844 raft_consensus.cc:2932] T 67ee1c64f81b492a9c83e7618878ff79 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:58:54.774626 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67ee1c64f81b492a9c83e7618878ff79 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:58:54.775573 434 catalog_manager.cc:5484] T 67ee1c64f81b492a9c83e7618878ff79 P ab254774096345abb0c1daf0f7afecac reported cstate change: config changed from index 7 to 8, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
W20240503 13:58:54.778380 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 67ee1c64f81b492a9c83e7618878ff79: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:58:54.803474 647 consensus_peers.cc:485] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:58:54.917698 909 consensus_peers.cc:485] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:58:55.015470 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 13:58:55.078810 976 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } }
I20240503 13:58:55.080121 713 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:58:55.080518 3217 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 13:58:55.080695 909 consensus_peers.cc:485] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:55.082088 3070 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } }
I20240503 13:58:55.082422 713 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } }
I20240503 13:58:55.084676 438 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:55.087565 976 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } }
I20240503 13:58:55.089146 713 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:55.089692 3217 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 13:58:55.090317 909 consensus_peers.cc:485] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:55.091369 3070 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } } }
I20240503 13:58:55.091746 713 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } } }
I20240503 13:58:55.093042 425 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 633e8264ce454183b2e0c0ad92f34629 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 13:58:55.093932 434 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index 4 to 5, NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:58:55.094662 912 consensus_peers.cc:485] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136:33227): Couldn't send request to peer 06de56aefa784b4c8b7d5a3e7c5f9825. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 633e8264ce454183b2e0c0ad92f34629. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:58:55.255703 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:58:55.257270 778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20240503 13:58:55.521644 3239 ts_tablet_manager.cc:902] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: Initiating tablet copy from peer e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:58:55.522239 3239 tablet_copy_client.cc:308] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40971
I20240503 13:58:55.525274 996 tablet_copy_service.cc:140] P e59e5d411bc046a28486330e3e4ffb66: Received BeginTabletCopySession request for tablet 633e8264ce454183b2e0c0ad92f34629 from peer 06de56aefa784b4c8b7d5a3e7c5f9825 ({username='slave'} at 127.0.99.136:58169)
I20240503 13:58:55.525364 996 tablet_copy_service.cc:161] P e59e5d411bc046a28486330e3e4ffb66: Beginning new tablet copy session on tablet 633e8264ce454183b2e0c0ad92f34629 from peer 06de56aefa784b4c8b7d5a3e7c5f9825 at {username='slave'} at 127.0.99.136:58169: session id = 06de56aefa784b4c8b7d5a3e7c5f9825-633e8264ce454183b2e0c0ad92f34629
I20240503 13:58:55.526070 996 tablet_copy_source_session.cc:205] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:55.528658 3239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633e8264ce454183b2e0c0ad92f34629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:55.533051 3239 tablet_copy_client.cc:783] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:55.533206 3239 tablet_copy_client.cc:647] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:55.534209 3243 tablet_copy_client.cc:1002] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Injecting latency into file download: 1500
W20240503 13:58:55.858721 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1d63d06c06ed4743b3e1176413142bef: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:58:56.170085 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:58:56.532723 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 13:58:56.984937 425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240503 13:58:56.985577 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 67ee1c64f81b492a9c83e7618878ff79: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:57.034587 3239 tablet_copy_client.cc:523] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:58:57.036818 3239 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: Bootstrap starting.
I20240503 13:58:57.042318 3239 log.cc:824] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: Log is configured to *not* fsync() on all Append() calls
W20240503 13:58:57.048147 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:58:57.061959 3239 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:58:57.062295 3239 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: Bootstrap complete.
I20240503 13:58:57.062549 3239 ts_tablet_manager.cc:1367] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240503 13:58:57.064285 3239 raft_consensus.cc:348] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } }
I20240503 13:58:57.064491 3239 raft_consensus.cc:729] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06de56aefa784b4c8b7d5a3e7c5f9825, State: Initialized, Role: LEARNER
I20240503 13:58:57.064812 3239 consensus_queue.cc:260] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } }
I20240503 13:58:57.065546 3239 ts_tablet_manager.cc:1398] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240503 13:58:57.066337 996 tablet_copy_service.cc:342] P e59e5d411bc046a28486330e3e4ffb66: Request end of tablet copy session 06de56aefa784b4c8b7d5a3e7c5f9825-633e8264ce454183b2e0c0ad92f34629 received from {username='slave'} at 127.0.99.136:58169
I20240503 13:58:57.066414 996 tablet_copy_service.cc:433] P e59e5d411bc046a28486330e3e4ffb66: ending tablet copy session 06de56aefa784b4c8b7d5a3e7c5f9825-633e8264ce454183b2e0c0ad92f34629 on tablet 633e8264ce454183b2e0c0ad92f34629 with peer 06de56aefa784b4c8b7d5a3e7c5f9825
W20240503 13:58:57.068645 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:58:57.094800 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:58:57.205386 778 consensus_peers.cc:485] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:58:57.236881 647 consensus_peers.cc:485] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:58:57.396735 2397 raft_consensus.cc:1203] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 13:58:57.548130 909 consensus_peers.cc:485] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 13:58:57.769464 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:58:57.777494 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 13:58:57.807201 3256 raft_consensus.cc:1050] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: attempting to promote NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 to VOTER
I20240503 13:58:57.807688 3256 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } }
I20240503 13:58:57.809005 714 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:58:57.809145 2397 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:58:57.809445 3309 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:58:57.809625 3311 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 13:58:57.809778 909 consensus_peers.cc:485] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:57.812224 3311 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:58:57.812729 2397 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:58:57.813901 714 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:58:57.815085 434 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 reported cstate change: config changed from index 5 to 6, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:58:57.818367 976 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } }
I20240503 13:58:57.819289 2397 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:58:57.819339 714 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:58:57.819662 3256 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:58:57.819764 3309 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:58:57.821177 3298 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:58:57.822288 425 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 633e8264ce454183b2e0c0ad92f34629 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:58:57.822551 2397 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:58:57.822664 714 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:58:57.823859 441 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:58:58.012629 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 633e8264ce454183b2e0c0ad92f34629: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:58:58.179751 844 consensus_queue.cc:237] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:58:58.180976 714 raft_consensus.cc:1261] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:58:58.181480 778 consensus_peers.cc:485] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:58.181550 3332 consensus_queue.cc:1035] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:58:58.183038 3032 raft_consensus.cc:2932] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } }
I20240503 13:58:58.183462 714 raft_consensus.cc:2932] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } }
I20240503 13:58:58.185590 434 catalog_manager.cc:5484] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } }
I20240503 13:58:58.188536 844 consensus_queue.cc:237] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
W20240503 13:58:58.190464 778 consensus_peers.cc:485] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:58.190539 714 raft_consensus.cc:1261] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:58:58.190968 3339 consensus_queue.cc:1035] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:58:58.192387 3332 raft_consensus.cc:2932] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } } }
I20240503 13:58:58.193918 427 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet eb3d68a0d39042c98d02c9a93307bb88 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 13:58:58.194521 434 catalog_manager.cc:5484] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index 4 to 5, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:58:58.195747 714 raft_consensus.cc:2932] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } } }
W20240503 13:58:58.196372 779 consensus_peers.cc:485] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 -> Peer 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134:36679): Couldn't send request to peer 3be34dcec5ad4ec7860fe6145ac58163. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb3d68a0d39042c98d02c9a93307bb88. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:58:58.608608 3354 ts_tablet_manager.cc:902] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: Initiating tablet copy from peer f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:58:58.609267 3354 tablet_copy_client.cc:308] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44047
I20240503 13:58:58.612614 865 tablet_copy_service.cc:140] P f58fb372531d42a3bc405510bf5bced4: Received BeginTabletCopySession request for tablet eb3d68a0d39042c98d02c9a93307bb88 from peer 3be34dcec5ad4ec7860fe6145ac58163 ({username='slave'} at 127.0.99.134:35791)
I20240503 13:58:58.612727 865 tablet_copy_service.cc:161] P f58fb372531d42a3bc405510bf5bced4: Beginning new tablet copy session on tablet eb3d68a0d39042c98d02c9a93307bb88 from peer 3be34dcec5ad4ec7860fe6145ac58163 at {username='slave'} at 127.0.99.134:35791: session id = 3be34dcec5ad4ec7860fe6145ac58163-eb3d68a0d39042c98d02c9a93307bb88
I20240503 13:58:58.613780 865 tablet_copy_source_session.cc:205] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:58:58.616719 3354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3d68a0d39042c98d02c9a93307bb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:58:58.621598 3354 tablet_copy_client.cc:783] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Starting download of 0 data blocks...
I20240503 13:58:58.621780 3354 tablet_copy_client.cc:647] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Starting download of 1 WAL segments...
I20240503 13:58:58.622998 3357 tablet_copy_client.cc:1002] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Injecting latency into file download: 1500
W20240503 13:58:58.776650 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:58:58.931154 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 13:58:59.061101 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 67ee1c64f81b492a9c83e7618878ff79: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:58:59.443037 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:58:59.494170 647 consensus_peers.cc:485] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:58:59.531319 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:58:59.736099 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:58:59.838438 909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 13:59:00.085072 647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 13:59:00.123456 3354 tablet_copy_client.cc:523] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:00.125300 3354 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap starting.
I20240503 13:59:00.130179 3354 log.cc:824] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:00.148027 3354 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:00.148325 3354 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap complete.
I20240503 13:59:00.148599 3354 ts_tablet_manager.cc:1367] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.004s
I20240503 13:59:00.150290 3354 raft_consensus.cc:348] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
I20240503 13:59:00.150475 3354 raft_consensus.cc:729] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3be34dcec5ad4ec7860fe6145ac58163, State: Initialized, Role: LEARNER
I20240503 13:59:00.150724 3354 consensus_queue.cc:260] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
I20240503 13:59:00.151329 3354 ts_tablet_manager.cc:1398] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:00.151926 865 tablet_copy_service.cc:342] P f58fb372531d42a3bc405510bf5bced4: Request end of tablet copy session 3be34dcec5ad4ec7860fe6145ac58163-eb3d68a0d39042c98d02c9a93307bb88 received from {username='slave'} at 127.0.99.134:35791
I20240503 13:59:00.152007 865 tablet_copy_service.cc:433] P f58fb372531d42a3bc405510bf5bced4: ending tablet copy session 3be34dcec5ad4ec7860fe6145ac58163-eb3d68a0d39042c98d02c9a93307bb88 on tablet eb3d68a0d39042c98d02c9a93307bb88 with peer 3be34dcec5ad4ec7860fe6145ac58163
W20240503 13:59:00.223513 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 13:59:00.375552 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 13:59:00.512071 2135 raft_consensus.cc:1203] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 13:59:00.638569 778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 13:59:00.639214 778 consensus_peers.cc:485] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 13:59:00.994354 3375 raft_consensus.cc:1050] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4: attempting to promote NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 to VOTER
I20240503 13:59:00.994777 3375 consensus_queue.cc:237] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } }
I20240503 13:59:00.996300 714 raft_consensus.cc:1261] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:00.996376 2135 raft_consensus.cc:1261] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163 [term 1 LEARNER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:00.996853 3431 consensus_queue.cc:1035] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 13:59:00.996989 778 consensus_peers.cc:485] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:00.997028 3375 consensus_queue.cc:1035] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:00.999577 3431 raft_consensus.cc:2932] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:01.000047 2135 raft_consensus.cc:2932] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:01.001495 714 raft_consensus.cc:2932] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:01.002974 438 catalog_manager.cc:5484] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163 reported cstate change: config changed from index 5 to 6, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:01.006474 844 consensus_queue.cc:237] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } }
I20240503 13:59:01.007604 714 raft_consensus.cc:1261] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:01.007658 2135 raft_consensus.cc:1261] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:01.008018 3375 consensus_queue.cc:1035] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:01.008778 3375 consensus_queue.cc:1035] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:01.009474 3431 raft_consensus.cc:2932] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:01.009941 714 raft_consensus.cc:2932] T eb3d68a0d39042c98d02c9a93307bb88 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:01.011160 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb3d68a0d39042c98d02c9a93307bb88 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:01.011715 441 catalog_manager.cc:5484] T eb3d68a0d39042c98d02c9a93307bb88 P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:01.012710 2135 raft_consensus.cc:2932] T eb3d68a0d39042c98d02c9a93307bb88 P 3be34dcec5ad4ec7860fe6145ac58163 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
W20240503 13:59:01.015846 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eb3d68a0d39042c98d02c9a93307bb88: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:59:01.032895 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 13:59:01.291107 714 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:59:01.292726 844 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:01.293181 3192 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:01.294932 3438 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } }
I20240503 13:59:01.295418 844 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } }
I20240503 13:59:01.297847 441 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } }
W20240503 13:59:01.298818 647 consensus_peers.cc:485] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:01.301278 714 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:01.302659 844 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:01.303069 3459 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 13:59:01.303529 647 consensus_peers.cc:485] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:01.304704 3192 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
I20240503 13:59:01.305065 844 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
I20240503 13:59:01.306401 428 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6e521d523c904d659b745874eec536e4 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 13:59:01.307199 441 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index 4 to 5, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
W20240503 13:59:01.309477 650 consensus_peers.cc:485] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 83070528d9574a528c6e57fd30b33dcc (127.0.99.133:34511): Couldn't send request to peer 83070528d9574a528c6e57fd30b33dcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e521d523c904d659b745874eec536e4. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:01.381822 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 13:59:01.701222 3469 ts_tablet_manager.cc:902] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: Initiating tablet copy from peer 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:59:01.701939 3469 tablet_copy_client.cc:308] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42209
I20240503 13:59:01.705333 734 tablet_copy_service.cc:140] P 37dbd0b8eaae46d59b318fbbb4bab576: Received BeginTabletCopySession request for tablet 6e521d523c904d659b745874eec536e4 from peer 83070528d9574a528c6e57fd30b33dcc ({username='slave'} at 127.0.99.133:59651)
I20240503 13:59:01.705435 734 tablet_copy_service.cc:161] P 37dbd0b8eaae46d59b318fbbb4bab576: Beginning new tablet copy session on tablet 6e521d523c904d659b745874eec536e4 from peer 83070528d9574a528c6e57fd30b33dcc at {username='slave'} at 127.0.99.133:59651: session id = 83070528d9574a528c6e57fd30b33dcc-6e521d523c904d659b745874eec536e4
I20240503 13:59:01.706212 734 tablet_copy_source_session.cc:205] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:01.709473 3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e521d523c904d659b745874eec536e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:01.714915 3469 tablet_copy_client.cc:783] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:01.715119 3469 tablet_copy_client.cc:647] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:01.716506 3473 tablet_copy_client.cc:1002] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Injecting latency into file download: 1500
W20240503 13:59:01.981473 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:59:02.107964 425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240503 13:59:02.108592 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 633e8264ce454183b2e0c0ad92f34629: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:59:02.266973 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:59:02.307834 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:59:02.608533 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:59:02.770141 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:59:03.188861 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 67ee1c64f81b492a9c83e7618878ff79: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:59:03.216953 3469 tablet_copy_client.cc:523] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:03.218959 3469 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: Bootstrap starting.
I20240503 13:59:03.224503 3469 log.cc:824] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:03.244297 3469 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:03.244652 3469 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: Bootstrap complete.
I20240503 13:59:03.244952 3469 ts_tablet_manager.cc:1367] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: Time spent bootstrapping tablet: real 0.026s user 0.026s sys 0.000s
I20240503 13:59:03.246898 3469 raft_consensus.cc:348] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:03.247125 3469 raft_consensus.cc:729] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83070528d9574a528c6e57fd30b33dcc, State: Initialized, Role: LEARNER
I20240503 13:59:03.247450 3469 consensus_queue.cc:260] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:03.248328 3469 ts_tablet_manager.cc:1398] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:03.249109 734 tablet_copy_service.cc:342] P 37dbd0b8eaae46d59b318fbbb4bab576: Request end of tablet copy session 83070528d9574a528c6e57fd30b33dcc-6e521d523c904d659b745874eec536e4 received from {username='slave'} at 127.0.99.133:59651
I20240503 13:59:03.249210 734 tablet_copy_service.cc:433] P 37dbd0b8eaae46d59b318fbbb4bab576: ending tablet copy session 83070528d9574a528c6e57fd30b33dcc-6e521d523c904d659b745874eec536e4 on tablet 6e521d523c904d659b745874eec536e4 with peer 83070528d9574a528c6e57fd30b33dcc
I20240503 13:59:03.422423 2004 raft_consensus.cc:1203] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 13:59:03.588016 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 13:59:03.672101 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 13:59:03.885277 3542 raft_consensus.cc:1050] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: attempting to promote NON_VOTER 83070528d9574a528c6e57fd30b33dcc to VOTER
I20240503 13:59:03.885673 3542 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:03.887058 844 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:03.887053 2004 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:03.887502 3542 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:03.887646 3541 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 13:59:03.888036 647 consensus_peers.cc:485] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:03.890067 3542 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:03.890583 2004 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:03.892153 844 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:03.893018 434 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: config changed from index 5 to 6, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:03.896405 713 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:03.897437 844 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:03.897430 2004 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:03.897784 3541 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:03.897939 3544 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:03.899482 3544 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:03.901105 2004 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:03.901125 844 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:03.902338 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e521d523c904d659b745874eec536e4 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:03.902937 438 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: config changed from index 6 to 7, VOTER 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:04.256565 956 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:59:04.258837 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 13:59:04.259131 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6e521d523c904d659b745874eec536e4: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:59:04.268424 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:04.270246 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:04.270785 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:04.279909 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:04.293334 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:04.376760 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:04.376935 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:04.379511 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:04.380983 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:04.382385 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:04.391345 824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:04.391892 956 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 13:59:04.437122 976 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } }
I20240503 13:59:04.438776 713 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:04.438783 844 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 13:59:04.438921 912 consensus_peers.cc:485] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136:33227): Couldn't send request to peer 06de56aefa784b4c8b7d5a3e7c5f9825. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0ecd805f87a4afeb6e41b1f6f671622. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:04.439285 3463 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:04.439486 3643 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:04.441287 3642 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } } }
I20240503 13:59:04.443027 713 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } } }
I20240503 13:59:04.445163 844 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } } }
I20240503 13:59:04.445374 432 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index -1 to 4, NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:04.454186 844 consensus_queue.cc:237] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:04.455693 976 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:04.456105 713 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:04.456467 3375 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:04.457140 3655 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:04.458348 3375 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
I20240503 13:59:04.459636 713 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
I20240503 13:59:04.460726 976 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
I20240503 13:59:04.461176 434 catalog_manager.cc:5484] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index -1 to 4, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:04.461427 781 consensus_peers.cc:485] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 -> Peer 83070528d9574a528c6e57fd30b33dcc (127.0.99.133:34511): Couldn't send request to peer 83070528d9574a528c6e57fd30b33dcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cc710cdf7be47dcbb95ccc49e5beabf. This is attempt 1: this message will repeat every 5th retry.
W20240503 13:59:04.648895 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:59:04.814733 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:59:04.827761 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:59:04.873345 3680 ts_tablet_manager.cc:902] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc: Initiating tablet copy from peer f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:59:04.873883 3680 tablet_copy_client.cc:308] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44047
I20240503 13:59:04.878175 865 tablet_copy_service.cc:140] P f58fb372531d42a3bc405510bf5bced4: Received BeginTabletCopySession request for tablet 3cc710cdf7be47dcbb95ccc49e5beabf from peer 83070528d9574a528c6e57fd30b33dcc ({username='slave'} at 127.0.99.133:54557)
I20240503 13:59:04.878284 865 tablet_copy_service.cc:161] P f58fb372531d42a3bc405510bf5bced4: Beginning new tablet copy session on tablet 3cc710cdf7be47dcbb95ccc49e5beabf from peer 83070528d9574a528c6e57fd30b33dcc at {username='slave'} at 127.0.99.133:54557: session id = 83070528d9574a528c6e57fd30b33dcc-3cc710cdf7be47dcbb95ccc49e5beabf
I20240503 13:59:04.878998 865 tablet_copy_source_session.cc:205] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:04.879807 3680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc710cdf7be47dcbb95ccc49e5beabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:04.881507 3680 tablet_copy_client.cc:783] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:04.881664 3680 tablet_copy_client.cc:647] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:04.882611 3682 tablet_copy_client.cc:1002] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Injecting latency into file download: 1500
I20240503 13:59:04.900125 3696 ts_tablet_manager.cc:902] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825: Initiating tablet copy from peer e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:59:04.900591 3696 tablet_copy_client.cc:308] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40971
I20240503 13:59:04.901067 996 tablet_copy_service.cc:140] P e59e5d411bc046a28486330e3e4ffb66: Received BeginTabletCopySession request for tablet b0ecd805f87a4afeb6e41b1f6f671622 from peer 06de56aefa784b4c8b7d5a3e7c5f9825 ({username='slave'} at 127.0.99.136:58169)
I20240503 13:59:04.901199 996 tablet_copy_service.cc:161] P e59e5d411bc046a28486330e3e4ffb66: Beginning new tablet copy session on tablet b0ecd805f87a4afeb6e41b1f6f671622 from peer 06de56aefa784b4c8b7d5a3e7c5f9825 at {username='slave'} at 127.0.99.136:58169: session id = 06de56aefa784b4c8b7d5a3e7c5f9825-b0ecd805f87a4afeb6e41b1f6f671622
I20240503 13:59:04.902146 996 tablet_copy_source_session.cc:205] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:04.903085 3696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ecd805f87a4afeb6e41b1f6f671622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:04.904882 3696 tablet_copy_client.cc:783] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:04.905028 3696 tablet_copy_client.cc:647] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:04.905917 3700 tablet_copy_client.cc:1002] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Injecting latency into file download: 1500
W20240503 13:59:05.028345 909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 13:59:05.028944 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:59:05.062443 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:59:05.407922 647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 13:59:06.130403 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6e521d523c904d659b745874eec536e4: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:59:06.204874 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 13:59:06.346112 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 13:59:06.382911 3680 tablet_copy_client.cc:523] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:06.384311 3680 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc: Bootstrap starting.
I20240503 13:59:06.399773 3680 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:06.400094 3680 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc: Bootstrap complete.
I20240503 13:59:06.400278 3680 ts_tablet_manager.cc:1367] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.004s
I20240503 13:59:06.400795 3680 raft_consensus.cc:348] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:06.400908 3680 raft_consensus.cc:729] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83070528d9574a528c6e57fd30b33dcc, State: Initialized, Role: LEARNER
I20240503 13:59:06.401008 3680 consensus_queue.cc:260] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:06.401342 3680 ts_tablet_manager.cc:1398] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:06.401985 865 tablet_copy_service.cc:342] P f58fb372531d42a3bc405510bf5bced4: Request end of tablet copy session 83070528d9574a528c6e57fd30b33dcc-3cc710cdf7be47dcbb95ccc49e5beabf received from {username='slave'} at 127.0.99.133:54557
I20240503 13:59:06.402063 865 tablet_copy_service.cc:433] P f58fb372531d42a3bc405510bf5bced4: ending tablet copy session 83070528d9574a528c6e57fd30b33dcc-3cc710cdf7be47dcbb95ccc49e5beabf on tablet 3cc710cdf7be47dcbb95ccc49e5beabf with peer 83070528d9574a528c6e57fd30b33dcc
I20240503 13:59:06.406188 3696 tablet_copy_client.cc:523] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:06.407505 3696 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825: Bootstrap starting.
I20240503 13:59:06.426101 3696 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:06.426401 3696 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825: Bootstrap complete.
I20240503 13:59:06.426610 3696 ts_tablet_manager.cc:1367] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.003s
I20240503 13:59:06.427227 3696 raft_consensus.cc:348] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } }
I20240503 13:59:06.427398 3696 raft_consensus.cc:729] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06de56aefa784b4c8b7d5a3e7c5f9825, State: Initialized, Role: LEARNER
I20240503 13:59:06.427493 3696 consensus_queue.cc:260] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } }
I20240503 13:59:06.427855 3696 ts_tablet_manager.cc:1398] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 13:59:06.428423 996 tablet_copy_service.cc:342] P e59e5d411bc046a28486330e3e4ffb66: Request end of tablet copy session 06de56aefa784b4c8b7d5a3e7c5f9825-b0ecd805f87a4afeb6e41b1f6f671622 received from {username='slave'} at 127.0.99.136:58169
I20240503 13:59:06.428517 996 tablet_copy_service.cc:433] P e59e5d411bc046a28486330e3e4ffb66: ending tablet copy session 06de56aefa784b4c8b7d5a3e7c5f9825-b0ecd805f87a4afeb6e41b1f6f671622 on tablet b0ecd805f87a4afeb6e41b1f6f671622 with peer 06de56aefa784b4c8b7d5a3e7c5f9825
I20240503 13:59:06.486872 2004 raft_consensus.cc:1203] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 13:59:06.936971 2397 raft_consensus.cc:1203] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 13:59:07.023530 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:59:07.078856 3822 raft_consensus.cc:1050] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: attempting to promote NON_VOTER 83070528d9574a528c6e57fd30b33dcc to VOTER
I20240503 13:59:07.079330 3822 consensus_queue.cc:237] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:07.080761 976 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:07.080761 2004 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:07.080806 713 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:07.081142 3824 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:07.081480 3821 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:07.081673 3822 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:07.083521 3824 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:07.083879 2004 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:07.083956 713 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:07.085440 976 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:07.086527 438 catalog_manager.cc:5484] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index 4 to 5, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:07.090075 844 consensus_queue.cc:237] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:07.091285 2004 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:07.091632 976 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:07.092033 3821 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:07.092235 3822 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:07.093410 3821 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:07.093852 976 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:07.093961 2004 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:07.095122 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3cc710cdf7be47dcbb95ccc49e5beabf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:07.095775 432 catalog_manager.cc:5484] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index 5 to 6, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:07.098109 693 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cc710cdf7be47dcbb95ccc49e5beabf with delete_type TABLET_DATA_TOMBSTONED (TS 37dbd0b8eaae46d59b318fbbb4bab576 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39974
I20240503 13:59:07.098603 3833 tablet_replica.cc:330] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576: stopping tablet replica
I20240503 13:59:07.098892 3833 raft_consensus.cc:2229] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:07.099092 3833 raft_consensus.cc:2258] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:07.100145 3833 ts_tablet_manager.cc:1875] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:07.102025 3833 ts_tablet_manager.cc:1888] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:07.102110 3833 log.cc:1196] T 3cc710cdf7be47dcbb95ccc49e5beabf P 37dbd0b8eaae46d59b318fbbb4bab576: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3cc710cdf7be47dcbb95ccc49e5beabf
I20240503 13:59:07.102648 428 catalog_manager.cc:4830] TS 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209): tablet 3cc710cdf7be47dcbb95ccc49e5beabf (table rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]) successfully deleted
I20240503 13:59:07.235692 976 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:07.237363 2397 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:07.237398 713 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:07.237795 3850 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:07.238003 3809 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:07.239828 3809 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
W20240503 13:59:07.240846 912 consensus_peers.cc:485] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 83070528d9574a528c6e57fd30b33dcc (127.0.99.133:34511): Couldn't send request to peer 83070528d9574a528c6e57fd30b33dcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 633e8264ce454183b2e0c0ad92f34629. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:07.241070 2397 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
I20240503 13:59:07.243487 437 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 reported cstate change: config changed from index 7 to 8, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
I20240503 13:59:07.244484 713 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
I20240503 13:59:07.251681 713 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:59:07.253284 844 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:07.253304 2004 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 13:59:07.253383 648 consensus_peers.cc:485] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer ab254774096345abb0c1daf0f7afecac (127.0.99.135:41905): Couldn't send request to peer ab254774096345abb0c1daf0f7afecac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e521d523c904d659b745874eec536e4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:07.253690 3542 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:07.253867 3861 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:07.255648 3542 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:59:07.256115 844 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:59:07.256157 2004 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:59:07.258502 437 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index 7 to 8, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
W20240503 13:59:07.273283 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:59:07.352646 3643 raft_consensus.cc:1050] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: attempting to promote NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 to VOTER
I20240503 13:59:07.353055 3643 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } }
I20240503 13:59:07.354455 2397 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:07.354475 844 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:07.354580 713 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:07.355090 3881 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:07.355262 3809 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:07.355422 3850 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:07.357240 3881 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:07.357661 844 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:07.359113 713 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:07.359674 2397 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:07.359974 432 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index 4 to 5, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:07.363142 976 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } }
I20240503 13:59:07.364183 2397 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:07.364259 713 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:07.364617 3881 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:07.364967 3881 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:07.366083 3809 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:07.367372 425 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0ecd805f87a4afeb6e41b1f6f671622 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:07.367774 713 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:07.368121 438 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index 5 to 6, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:07.368146 2397 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:07.371038 824 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ecd805f87a4afeb6e41b1f6f671622 with delete_type TABLET_DATA_TOMBSTONED (TS f58fb372531d42a3bc405510bf5bced4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45588
I20240503 13:59:07.371665 3885 tablet_replica.cc:330] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4: stopping tablet replica
I20240503 13:59:07.372016 3885 raft_consensus.cc:2229] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:07.372238 3885 raft_consensus.cc:2258] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:07.373229 3885 ts_tablet_manager.cc:1875] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:07.375047 3885 ts_tablet_manager.cc:1888] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:07.375126 3885 log.cc:1196] T b0ecd805f87a4afeb6e41b1f6f671622 P f58fb372531d42a3bc405510bf5bced4: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b0ecd805f87a4afeb6e41b1f6f671622
I20240503 13:59:07.375681 427 catalog_manager.cc:4830] TS f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047): tablet b0ecd805f87a4afeb6e41b1f6f671622 (table rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]) successfully deleted
W20240503 13:59:07.489010 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:59:07.503690 976 consensus_queue.cc:237] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
I20240503 13:59:07.505242 844 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 13:59:07.505467 2266 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 13:59:07.505661 3881 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:07.506129 3881 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:07.507436 3850 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } } }
I20240503 13:59:07.510267 437 catalog_manager.cc:5484] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index 8 to 9, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:07.510602 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:59:07.511518 910 consensus_peers.cc:485] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 -> Peer 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134:36679): Couldn't send request to peer 3be34dcec5ad4ec7860fe6145ac58163. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d63d06c06ed4743b3e1176413142bef. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:07.511691 2266 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } } }
I20240503 13:59:07.512112 844 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } } }
W20240503 13:59:07.595500 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:59:07.803340 3943 ts_tablet_manager.cc:902] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc: Initiating tablet copy from peer e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:59:07.803886 3943 tablet_copy_client.cc:308] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40971
I20240503 13:59:07.807298 996 tablet_copy_service.cc:140] P e59e5d411bc046a28486330e3e4ffb66: Received BeginTabletCopySession request for tablet 633e8264ce454183b2e0c0ad92f34629 from peer 83070528d9574a528c6e57fd30b33dcc ({username='slave'} at 127.0.99.133:54493)
I20240503 13:59:07.807422 996 tablet_copy_service.cc:161] P e59e5d411bc046a28486330e3e4ffb66: Beginning new tablet copy session on tablet 633e8264ce454183b2e0c0ad92f34629 from peer 83070528d9574a528c6e57fd30b33dcc at {username='slave'} at 127.0.99.133:54493: session id = 83070528d9574a528c6e57fd30b33dcc-633e8264ce454183b2e0c0ad92f34629
I20240503 13:59:07.808434 996 tablet_copy_source_session.cc:205] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:07.809307 3943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633e8264ce454183b2e0c0ad92f34629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.811105 3943 tablet_copy_client.cc:783] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:07.811287 3943 tablet_copy_client.cc:647] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:07.812260 3947 tablet_copy_client.cc:1002] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Injecting latency into file download: 1500
I20240503 13:59:07.863066 3949 ts_tablet_manager.cc:902] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac: Initiating tablet copy from peer 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:59:07.863716 3949 tablet_copy_client.cc:308] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42209
I20240503 13:59:07.864356 734 tablet_copy_service.cc:140] P 37dbd0b8eaae46d59b318fbbb4bab576: Received BeginTabletCopySession request for tablet 6e521d523c904d659b745874eec536e4 from peer ab254774096345abb0c1daf0f7afecac ({username='slave'} at 127.0.99.135:57933)
I20240503 13:59:07.864485 734 tablet_copy_service.cc:161] P 37dbd0b8eaae46d59b318fbbb4bab576: Beginning new tablet copy session on tablet 6e521d523c904d659b745874eec536e4 from peer ab254774096345abb0c1daf0f7afecac at {username='slave'} at 127.0.99.135:57933: session id = ab254774096345abb0c1daf0f7afecac-6e521d523c904d659b745874eec536e4
I20240503 13:59:07.865216 734 tablet_copy_source_session.cc:205] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:07.866161 3949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e521d523c904d659b745874eec536e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.868146 3949 tablet_copy_client.cc:783] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:07.868296 3949 tablet_copy_client.cc:647] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:07.869271 3953 tablet_copy_client.cc:1002] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac: tablet copy: Injecting latency into file download: 1500
I20240503 13:59:07.941480 3954 ts_tablet_manager.cc:902] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163: Initiating tablet copy from peer e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:59:07.942093 3954 tablet_copy_client.cc:308] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40971
I20240503 13:59:07.945712 996 tablet_copy_service.cc:140] P e59e5d411bc046a28486330e3e4ffb66: Received BeginTabletCopySession request for tablet 1d63d06c06ed4743b3e1176413142bef from peer 3be34dcec5ad4ec7860fe6145ac58163 ({username='slave'} at 127.0.99.134:44933)
I20240503 13:59:07.945802 996 tablet_copy_service.cc:161] P e59e5d411bc046a28486330e3e4ffb66: Beginning new tablet copy session on tablet 1d63d06c06ed4743b3e1176413142bef from peer 3be34dcec5ad4ec7860fe6145ac58163 at {username='slave'} at 127.0.99.134:44933: session id = 3be34dcec5ad4ec7860fe6145ac58163-1d63d06c06ed4743b3e1176413142bef
I20240503 13:59:07.946673 996 tablet_copy_source_session.cc:205] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:07.947780 3954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d63d06c06ed4743b3e1176413142bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:07.949523 3954 tablet_copy_client.cc:783] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:07.949681 3954 tablet_copy_client.cc:647] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:07.950726 3958 tablet_copy_client.cc:1002] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Injecting latency into file download: 1500
W20240503 13:59:08.188133 425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240503 13:59:08.188822 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1d63d06c06ed4743b3e1176413142bef: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:59:08.821283 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 13:59:08.982357 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 13:59:09.312551 3943 tablet_copy_client.cc:523] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:09.313922 3943 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc: Bootstrap starting.
I20240503 13:59:09.331755 3943 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:09.332046 3943 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc: Bootstrap complete.
I20240503 13:59:09.332233 3943 ts_tablet_manager.cc:1367] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.001s
I20240503 13:59:09.332646 3943 raft_consensus.cc:348] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:09.332763 3943 raft_consensus.cc:729] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83070528d9574a528c6e57fd30b33dcc, State: Initialized, Role: LEARNER
I20240503 13:59:09.332846 3943 consensus_queue.cc:260] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:09.333115 3943 ts_tablet_manager.cc:1398] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:09.333724 996 tablet_copy_service.cc:342] P e59e5d411bc046a28486330e3e4ffb66: Request end of tablet copy session 83070528d9574a528c6e57fd30b33dcc-633e8264ce454183b2e0c0ad92f34629 received from {username='slave'} at 127.0.99.133:54493
I20240503 13:59:09.333807 996 tablet_copy_service.cc:433] P e59e5d411bc046a28486330e3e4ffb66: ending tablet copy session 83070528d9574a528c6e57fd30b33dcc-633e8264ce454183b2e0c0ad92f34629 on tablet 633e8264ce454183b2e0c0ad92f34629 with peer 83070528d9574a528c6e57fd30b33dcc
I20240503 13:59:09.369664 3949 tablet_copy_client.cc:523] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:09.371052 3949 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac: Bootstrap starting.
W20240503 13:59:09.380612 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eb3d68a0d39042c98d02c9a93307bb88: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:59:09.390497 3949 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:09.390785 3949 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac: Bootstrap complete.
I20240503 13:59:09.390967 3949 ts_tablet_manager.cc:1367] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240503 13:59:09.391410 3949 raft_consensus.cc:348] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:59:09.391521 3949 raft_consensus.cc:729] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab254774096345abb0c1daf0f7afecac, State: Initialized, Role: LEARNER
I20240503 13:59:09.391620 3949 consensus_queue.cc:260] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:59:09.391923 3949 ts_tablet_manager.cc:1398] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 13:59:09.392486 734 tablet_copy_service.cc:342] P 37dbd0b8eaae46d59b318fbbb4bab576: Request end of tablet copy session ab254774096345abb0c1daf0f7afecac-6e521d523c904d659b745874eec536e4 received from {username='slave'} at 127.0.99.135:57933
I20240503 13:59:09.392580 734 tablet_copy_service.cc:433] P 37dbd0b8eaae46d59b318fbbb4bab576: ending tablet copy session ab254774096345abb0c1daf0f7afecac-6e521d523c904d659b745874eec536e4 on tablet 6e521d523c904d659b745874eec536e4 with peer ab254774096345abb0c1daf0f7afecac
I20240503 13:59:09.439800 2266 raft_consensus.cc:1203] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 13:59:09.451117 3954 tablet_copy_client.cc:523] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:09.452481 3954 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap starting.
I20240503 13:59:09.473664 3954 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:09.473984 3954 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap complete.
I20240503 13:59:09.474171 3954 ts_tablet_manager.cc:1367] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240503 13:59:09.474730 3954 raft_consensus.cc:348] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
I20240503 13:59:09.474862 3954 raft_consensus.cc:729] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3be34dcec5ad4ec7860fe6145ac58163, State: Initialized, Role: LEARNER
I20240503 13:59:09.475023 3954 consensus_queue.cc:260] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
I20240503 13:59:09.475500 3954 ts_tablet_manager.cc:1398] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 13:59:09.476066 996 tablet_copy_service.cc:342] P e59e5d411bc046a28486330e3e4ffb66: Request end of tablet copy session 3be34dcec5ad4ec7860fe6145ac58163-1d63d06c06ed4743b3e1176413142bef received from {username='slave'} at 127.0.99.134:44933
I20240503 13:59:09.476136 996 tablet_copy_service.cc:433] P e59e5d411bc046a28486330e3e4ffb66: ending tablet copy session 3be34dcec5ad4ec7860fe6145ac58163-1d63d06c06ed4743b3e1176413142bef on tablet 1d63d06c06ed4743b3e1176413142bef with peer 3be34dcec5ad4ec7860fe6145ac58163
W20240503 13:59:09.621369 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:59:09.814455 2135 raft_consensus.cc:1203] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240503 13:59:09.868916 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:59:09.881460 2004 raft_consensus.cc:1203] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 13:59:09.928071 4045 raft_consensus.cc:1050] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: attempting to promote NON_VOTER ab254774096345abb0c1daf0f7afecac to VOTER
I20240503 13:59:09.928565 4045 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:09.929845 2266 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 1 LEARNER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:09.930083 844 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:09.930224 4137 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:09.930418 2004 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:09.930464 4134 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:09.930805 4137 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:09.932240 4134 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:09.932691 2266 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:09.934485 844 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:09.934813 437 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac reported cstate change: config changed from index 8 to 9, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:09.936477 2004 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
W20240503 13:59:10.068166 909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 13:59:10.068782 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:59:10.130407 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:59:10.137358 713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e521d523c904d659b745874eec536e4"
dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35458
I20240503 13:59:10.137532 713 raft_consensus.cc:595] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:10.182415 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:59:10.358683 713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e521d523c904d659b745874eec536e4"
dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35462
I20240503 13:59:10.358870 713 raft_consensus.cc:595] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:10.368674 4186 raft_consensus.cc:1050] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66: attempting to promote NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 to VOTER
I20240503 13:59:10.369182 4186 consensus_queue.cc:237] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } }
I20240503 13:59:10.370508 843 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:10.370631 2135 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:10.370707 2266 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:10.370934 4137 raft_consensus.cc:979] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: : Instructing follower f58fb372531d42a3bc405510bf5bced4 to start an election
I20240503 13:59:10.371029 4137 raft_consensus.cc:1067] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Signalling peer f58fb372531d42a3bc405510bf5bced4 to start an election
I20240503 13:59:10.371016 4188 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:10.371191 4185 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:10.371434 4102 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:10.371660 843 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e521d523c904d659b745874eec536e4"
dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
from {username='slave'} at 127.0.99.130:48163
I20240503 13:59:10.371820 843 raft_consensus.cc:482] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:10.371889 843 raft_consensus.cc:3037] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:10.372906 843 raft_consensus.cc:504] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:10.373385 843 leader_election.cc:288] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 2 election: Requested vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), 83070528d9574a528c6e57fd30b33dcc (127.0.99.133:34511), ab254774096345abb0c1daf0f7afecac (127.0.99.135:41905)
I20240503 13:59:10.373770 843 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:10.373872 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e521d523c904d659b745874eec536e4" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
I20240503 13:59:10.373986 713 raft_consensus.cc:3032] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:10.374033 713 raft_consensus.cc:729] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Running, Role: LEADER
I20240503 13:59:10.374176 713 consensus_queue.cc:260] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:10.374372 713 raft_consensus.cc:3037] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:10.375171 713 raft_consensus.cc:2454] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58fb372531d42a3bc405510bf5bced4 in term 2.
I20240503 13:59:10.375185 2135 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:10.375526 2004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e521d523c904d659b745874eec536e4" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "83070528d9574a528c6e57fd30b33dcc"
I20240503 13:59:10.375666 2004 raft_consensus.cc:3037] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:10.376534 2004 raft_consensus.cc:2454] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58fb372531d42a3bc405510bf5bced4 in term 2.
I20240503 13:59:10.376927 781 leader_election.cc:302] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 37dbd0b8eaae46d59b318fbbb4bab576, 83070528d9574a528c6e57fd30b33dcc, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:59:10.373364 4188 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:10.377594 438 catalog_manager.cc:5484] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163 reported cstate change: config changed from index 9 to 10, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:10.378609 2266 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:10.379163 3755 raft_consensus.cc:2781] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:10.379338 3755 raft_consensus.cc:686] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 LEADER]: Becoming Leader. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Running, Role: LEADER
I20240503 13:59:10.379468 3755 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:10.381675 976 consensus_queue.cc:237] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } }
I20240503 13:59:10.382927 2135 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 13:59:10.383000 2266 raft_consensus.cc:1261] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 13:59:10.382948 434 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 reported cstate change: term changed from 1 to 2, leader changed from 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) to f58fb372531d42a3bc405510bf5bced4 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:10.383404 4186 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:10.383639 4188 consensus_queue.cc:1035] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:10.385062 4186 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:10.385784 2135 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:10.386759 425 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d63d06c06ed4743b3e1176413142bef with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:10.387768 434 catalog_manager.cc:5484] T 1d63d06c06ed4743b3e1176413142bef P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index 10 to 11, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:10.388450 2266 raft_consensus.cc:2932] T 1d63d06c06ed4743b3e1176413142bef P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:10.389001 2265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e521d523c904d659b745874eec536e4" candidate_uuid: "f58fb372531d42a3bc405510bf5bced4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "ab254774096345abb0c1daf0f7afecac"
I20240503 13:59:10.389168 2265 raft_consensus.cc:3037] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:10.390259 2265 raft_consensus.cc:2454] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58fb372531d42a3bc405510bf5bced4 in term 2.
I20240503 13:59:10.390261 824 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d63d06c06ed4743b3e1176413142bef with delete_type TABLET_DATA_TOMBSTONED (TS f58fb372531d42a3bc405510bf5bced4 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:45588
I20240503 13:59:10.391096 4196 tablet_replica.cc:330] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4: stopping tablet replica
I20240503 13:59:10.391270 4196 raft_consensus.cc:2229] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:10.391628 4196 raft_consensus.cc:2258] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:10.392222 4196 ts_tablet_manager.cc:1875] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:10.393838 4196 ts_tablet_manager.cc:1888] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 13:59:10.393898 4196 log.cc:1196] T 1d63d06c06ed4743b3e1176413142bef P f58fb372531d42a3bc405510bf5bced4: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1d63d06c06ed4743b3e1176413142bef
I20240503 13:59:10.394343 427 catalog_manager.cc:4830] TS f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047): tablet 1d63d06c06ed4743b3e1176413142bef (table rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]) successfully deleted
I20240503 13:59:10.451269 4102 raft_consensus.cc:1050] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: attempting to promote NON_VOTER 83070528d9574a528c6e57fd30b33dcc to VOTER
I20240503 13:59:10.451663 4102 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:10.453192 713 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:10.453205 2004 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:10.453231 2397 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 13:59:10.453606 4102 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:10.453840 4188 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:10.454104 4185 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:10.455837 4188 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:10.456250 2004 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:10.456347 2397 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:10.458349 713 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:10.458901 438 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index 8 to 9, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:10.586323 976 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "633e8264ce454183b2e0c0ad92f34629"
dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34034
I20240503 13:59:10.586484 976 raft_consensus.cc:595] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:10.593745 713 consensus_queue.cc:237] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } }
I20240503 13:59:10.595305 843 raft_consensus.cc:1261] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:10.595614 976 raft_consensus.cc:1261] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 13:59:10.595716 4135 consensus_queue.cc:1035] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:10.595958 4137 consensus_queue.cc:1035] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:10.597566 4135 raft_consensus.cc:2932] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } } }
I20240503 13:59:10.598055 843 raft_consensus.cc:2932] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } } }
I20240503 13:59:10.598502 976 raft_consensus.cc:2932] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } } }
I20240503 13:59:10.600435 434 catalog_manager.cc:5484] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: config changed from index -1 to 4, NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:10.602671 650 consensus_peers.cc:485] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136:33227): Couldn't send request to peer 06de56aefa784b4c8b7d5a3e7c5f9825. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f90e135e32343dfb01a1e435e87bd3e. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:10.623654 976 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "633e8264ce454183b2e0c0ad92f34629"
dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34054
I20240503 13:59:10.623780 976 raft_consensus.cc:595] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:10.688082 647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 13:59:10.822575 713 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:10.823026 3755 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:10.848084 976 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "633e8264ce454183b2e0c0ad92f34629"
dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34084
I20240503 13:59:10.848210 976 raft_consensus.cc:595] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:10.889209 2004 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:10.889694 3755 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:10.892268 2266 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:10.892887 4276 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:10.933935 4186 raft_consensus.cc:979] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: : Instructing follower 06de56aefa784b4c8b7d5a3e7c5f9825 to start an election
I20240503 13:59:10.934079 4186 raft_consensus.cc:1067] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Signalling peer 06de56aefa784b4c8b7d5a3e7c5f9825 to start an election
I20240503 13:59:10.934631 2397 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "633e8264ce454183b2e0c0ad92f34629"
dest_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825"
from {username='slave'} at 127.0.99.132:36619
I20240503 13:59:10.934826 2397 raft_consensus.cc:482] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:10.934938 2397 raft_consensus.cc:3037] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:10.935892 2397 raft_consensus.cc:504] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:10.936919 2397 leader_election.cc:288] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [CANDIDATE]: Term 2 election: Requested vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971), 83070528d9574a528c6e57fd30b33dcc (127.0.99.133:34511)
I20240503 13:59:10.937247 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633e8264ce454183b2e0c0ad92f34629" candidate_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
I20240503 13:59:10.937366 976 raft_consensus.cc:3032] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:10.937405 976 raft_consensus.cc:729] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Running, Role: LEADER
I20240503 13:59:10.937564 976 consensus_queue.cc:260] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:10.937788 976 raft_consensus.cc:3037] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:10.938836 976 raft_consensus.cc:2454] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06de56aefa784b4c8b7d5a3e7c5f9825 in term 2.
I20240503 13:59:10.940995 2004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633e8264ce454183b2e0c0ad92f34629" candidate_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "83070528d9574a528c6e57fd30b33dcc"
I20240503 13:59:10.941007 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633e8264ce454183b2e0c0ad92f34629" candidate_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
I20240503 13:59:10.941128 713 raft_consensus.cc:3037] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:10.941136 2004 raft_consensus.cc:3037] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:10.942077 713 raft_consensus.cc:2454] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06de56aefa784b4c8b7d5a3e7c5f9825 in term 2.
I20240503 13:59:10.942077 2004 raft_consensus.cc:2454] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06de56aefa784b4c8b7d5a3e7c5f9825 in term 2.
I20240503 13:59:10.942397 2333 leader_election.cc:302] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 06de56aefa784b4c8b7d5a3e7c5f9825, 83070528d9574a528c6e57fd30b33dcc, e59e5d411bc046a28486330e3e4ffb66; no voters:
I20240503 13:59:10.942646 4201 raft_consensus.cc:2781] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:10.942853 4201 raft_consensus.cc:686] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 LEADER]: Becoming Leader. State: Replica: 06de56aefa784b4c8b7d5a3e7c5f9825, State: Running, Role: LEADER
I20240503 13:59:10.943161 4201 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:10.945590 432 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 reported cstate change: term changed from 1 to 2, leader changed from e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132) to 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:11.125182 4300 ts_tablet_manager.cc:902] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825: Initiating tablet copy from peer 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:59:11.125676 4300 tablet_copy_client.cc:308] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42209
I20240503 13:59:11.126178 734 tablet_copy_service.cc:140] P 37dbd0b8eaae46d59b318fbbb4bab576: Received BeginTabletCopySession request for tablet 2f90e135e32343dfb01a1e435e87bd3e from peer 06de56aefa784b4c8b7d5a3e7c5f9825 ({username='slave'} at 127.0.99.136:41271)
I20240503 13:59:11.126282 734 tablet_copy_service.cc:161] P 37dbd0b8eaae46d59b318fbbb4bab576: Beginning new tablet copy session on tablet 2f90e135e32343dfb01a1e435e87bd3e from peer 06de56aefa784b4c8b7d5a3e7c5f9825 at {username='slave'} at 127.0.99.136:41271: session id = 06de56aefa784b4c8b7d5a3e7c5f9825-2f90e135e32343dfb01a1e435e87bd3e
I20240503 13:59:11.127039 734 tablet_copy_source_session.cc:205] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:11.127915 4300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f90e135e32343dfb01a1e435e87bd3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:11.129838 4300 tablet_copy_client.cc:783] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:11.130091 4300 tablet_copy_client.cc:647] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:11.131094 4304 tablet_copy_client.cc:1002] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Injecting latency into file download: 1500
W20240503 13:59:11.135072 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 13:59:11.183705 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 13:59:11.324909 976 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:11.325410 4201 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:11.401718 843 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:11.402807 2266 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 13:59:11.402839 2004 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 13:59:11.403191 4322 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:11.403390 4323 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:11.404865 4276 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:11.405300 2004 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:11.406921 427 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e521d523c904d659b745874eec536e4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:11.406911 2266 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:11.407752 432 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc reported cstate change: config changed from index 9 to 11, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:11.410327 693 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e521d523c904d659b745874eec536e4 with delete_type TABLET_DATA_TOMBSTONED (TS 37dbd0b8eaae46d59b318fbbb4bab576 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:39974
I20240503 13:59:11.410648 4326 tablet_replica.cc:330] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: stopping tablet replica
I20240503 13:59:11.410781 4326 raft_consensus.cc:2229] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:11.410904 4326 raft_consensus.cc:2258] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:11.411480 4326 ts_tablet_manager.cc:1875] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:11.413158 4326 ts_tablet_manager.cc:1888] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 13:59:11.413225 4326 log.cc:1196] T 6e521d523c904d659b745874eec536e4 P 37dbd0b8eaae46d59b318fbbb4bab576: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6e521d523c904d659b745874eec536e4
I20240503 13:59:11.413682 428 catalog_manager.cc:4830] TS 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209): tablet 6e521d523c904d659b745874eec536e4 (table rebalance_test_table_4 [id=d28850114dae4436ae20500518f1e627]) successfully deleted
W20240503 13:59:11.425580 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 67ee1c64f81b492a9c83e7618878ff79: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
I20240503 13:59:11.443361 713 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:11.443825 4201 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:11.446549 2004 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:11.447113 4329 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:11.936507 2397 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:11.937644 2004 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 13:59:11.937703 714 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 13:59:11.938040 4331 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:11.938258 4329 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:11.939795 4201 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:11.940172 2004 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:11.941824 714 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:11.942515 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 633e8264ce454183b2e0c0ad92f34629 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:11.942812 434 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc reported cstate change: config changed from index 9 to 11, VOTER e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:11.945593 956 tablet_service.cc:1514] Processing DeleteTablet for tablet 633e8264ce454183b2e0c0ad92f34629 with delete_type TABLET_DATA_TOMBSTONED (TS e59e5d411bc046a28486330e3e4ffb66 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41916
I20240503 13:59:11.946043 4390 tablet_replica.cc:330] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: stopping tablet replica
I20240503 13:59:11.946327 4390 raft_consensus.cc:2229] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:11.946456 4390 raft_consensus.cc:2258] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:11.947393 4390 ts_tablet_manager.cc:1875] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:11.949247 4390 ts_tablet_manager.cc:1888] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 13:59:11.949319 4390 log.cc:1196] T 633e8264ce454183b2e0c0ad92f34629 P e59e5d411bc046a28486330e3e4ffb66: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/633e8264ce454183b2e0c0ad92f34629
I20240503 13:59:11.949858 425 catalog_manager.cc:4830] TS e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971): tablet 633e8264ce454183b2e0c0ad92f34629 (table rebalance_test_table_0 [id=6fb6addecd294243a4c390358564540d]) successfully deleted
W20240503 13:59:12.255951 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:59:12.590904 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:59:12.631440 4300 tablet_copy_client.cc:523] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:12.632725 4300 tablet_bootstrap.cc:492] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825: Bootstrap starting.
I20240503 13:59:12.651120 4300 tablet_bootstrap.cc:492] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:12.651428 4300 tablet_bootstrap.cc:492] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825: Bootstrap complete.
I20240503 13:59:12.651579 4300 ts_tablet_manager.cc:1367] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.004s
I20240503 13:59:12.651917 4300 raft_consensus.cc:348] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } }
I20240503 13:59:12.652017 4300 raft_consensus.cc:729] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06de56aefa784b4c8b7d5a3e7c5f9825, State: Initialized, Role: LEARNER
I20240503 13:59:12.652093 4300 consensus_queue.cc:260] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: true } }
I20240503 13:59:12.652354 4300 ts_tablet_manager.cc:1398] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:12.652873 734 tablet_copy_service.cc:342] P 37dbd0b8eaae46d59b318fbbb4bab576: Request end of tablet copy session 06de56aefa784b4c8b7d5a3e7c5f9825-2f90e135e32343dfb01a1e435e87bd3e received from {username='slave'} at 127.0.99.136:41271
I20240503 13:59:12.652966 734 tablet_copy_service.cc:433] P 37dbd0b8eaae46d59b318fbbb4bab576: ending tablet copy session 06de56aefa784b4c8b7d5a3e7c5f9825-2f90e135e32343dfb01a1e435e87bd3e on tablet 2f90e135e32343dfb01a1e435e87bd3e with peer 06de56aefa784b4c8b7d5a3e7c5f9825
W20240503 13:59:12.672742 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:59:12.706063 2397 raft_consensus.cc:1203] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 13:59:12.849929 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:59:12.990639 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:59:13.113447 4433 raft_consensus.cc:1050] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576: attempting to promote NON_VOTER 06de56aefa784b4c8b7d5a3e7c5f9825 to VOTER
I20240503 13:59:13.113935 4433 consensus_queue.cc:237] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } }
I20240503 13:59:13.115532 976 raft_consensus.cc:1261] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:13.115550 843 raft_consensus.cc:1261] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:13.115903 4433 consensus_queue.cc:1035] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:13.116061 2397 raft_consensus.cc:1261] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 LEARNER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 13:59:13.116150 4435 consensus_queue.cc:1035] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:13.116423 4432 consensus_queue.cc:1035] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:13.118031 4432 raft_consensus.cc:2932] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:13.119061 2397 raft_consensus.cc:2932] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:13.119520 976 raft_consensus.cc:2932] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:13.121582 432 catalog_manager.cc:5484] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825 reported cstate change: config changed from index 4 to 5, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:13.121827 843 raft_consensus.cc:2932] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:13.125967 714 consensus_queue.cc:237] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } }
I20240503 13:59:13.126910 2397 raft_consensus.cc:1261] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:13.126909 976 raft_consensus.cc:1261] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 13:59:13.127318 4435 consensus_queue.cc:1035] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:13.127497 4432 consensus_queue.cc:1035] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 13:59:13.128829 4361 raft_consensus.cc:2932] T 2f90e135e32343dfb01a1e435e87bd3e P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:13.129253 976 raft_consensus.cc:2932] T 2f90e135e32343dfb01a1e435e87bd3e P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:13.129320 2397 raft_consensus.cc:2932] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:13.130704 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f90e135e32343dfb01a1e435e87bd3e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:13.131618 434 catalog_manager.cc:5484] T 2f90e135e32343dfb01a1e435e87bd3e P 06de56aefa784b4c8b7d5a3e7c5f9825 reported cstate change: config changed from index 5 to 6, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } }
I20240503 13:59:13.133635 824 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f90e135e32343dfb01a1e435e87bd3e with delete_type TABLET_DATA_TOMBSTONED (TS f58fb372531d42a3bc405510bf5bced4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45588
I20240503 13:59:13.133955 4443 tablet_replica.cc:330] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4: stopping tablet replica
I20240503 13:59:13.134083 4443 raft_consensus.cc:2229] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:13.134315 4443 raft_consensus.cc:2258] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:13.134943 4443 ts_tablet_manager.cc:1875] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:13.136777 4443 ts_tablet_manager.cc:1888] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 13:59:13.136847 4443 log.cc:1196] T 2f90e135e32343dfb01a1e435e87bd3e P f58fb372531d42a3bc405510bf5bced4: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2f90e135e32343dfb01a1e435e87bd3e
I20240503 13:59:13.137321 427 catalog_manager.cc:4830] TS f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047): tablet 2f90e135e32343dfb01a1e435e87bd3e (table rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]) successfully deleted
I20240503 13:59:13.237280 2377 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:13.240500 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:13.241235 1984 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:13.242450 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:13.252333 2115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:13.266979 956 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 13:59:13.279273 824 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:13.306267 2397 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
I20240503 13:59:13.307914 714 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 13:59:13.308248 2004 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 13:59:13.308444 4331 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:13.308848 4495 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:13.310446 4331 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } } }
I20240503 13:59:13.310868 714 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } } }
I20240503 13:59:13.313194 437 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 reported cstate change: config changed from index 11 to 12, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:13.313738 2331 consensus_peers.cc:485] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 -> Peer 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134:36679): Couldn't send request to peer 3be34dcec5ad4ec7860fe6145ac58163. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 633e8264ce454183b2e0c0ad92f34629. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:13.314533 2004 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } } }
I20240503 13:59:13.323117 976 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:13.324586 714 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:13.324633 2397 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 13:59:13.324735 912 consensus_peers.cc:485] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 83070528d9574a528c6e57fd30b33dcc (127.0.99.133:34511): Couldn't send request to peer 83070528d9574a528c6e57fd30b33dcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0ecd805f87a4afeb6e41b1f6f671622. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:13.324967 4507 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:13.325141 4508 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:13.326725 4186 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
I20240503 13:59:13.327111 714 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
I20240503 13:59:13.327414 2397 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } } }
I20240503 13:59:13.329449 437 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index 6 to 7, NON_VOTER 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:13.337904 843 consensus_queue.cc:237] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:59:13.339637 2004 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 13:59:13.339741 976 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 13:59:13.339952 779 consensus_peers.cc:485] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 -> Peer ab254774096345abb0c1daf0f7afecac (127.0.99.135:41905): Couldn't send request to peer ab254774096345abb0c1daf0f7afecac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cc710cdf7be47dcbb95ccc49e5beabf. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:13.340111 4519 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:13.340281 4322 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 13:59:13.342366 976 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:59:13.342414 2004 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:59:13.341964 4518 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:59:13.344686 434 catalog_manager.cc:5484] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 reported cstate change: config changed from index 6 to 7, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
W20240503 13:59:13.532613 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 13:59:13.732257 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 13:59:13.791427 4563 ts_tablet_manager.cc:902] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163: Initiating tablet copy from peer 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136:33227)
I20240503 13:59:13.791832 4563 tablet_copy_client.cc:308] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:33227
I20240503 13:59:13.795603 2417 tablet_copy_service.cc:140] P 06de56aefa784b4c8b7d5a3e7c5f9825: Received BeginTabletCopySession request for tablet 633e8264ce454183b2e0c0ad92f34629 from peer 3be34dcec5ad4ec7860fe6145ac58163 ({username='slave'} at 127.0.99.134:40663)
I20240503 13:59:13.795727 2417 tablet_copy_service.cc:161] P 06de56aefa784b4c8b7d5a3e7c5f9825: Beginning new tablet copy session on tablet 633e8264ce454183b2e0c0ad92f34629 from peer 3be34dcec5ad4ec7860fe6145ac58163 at {username='slave'} at 127.0.99.134:40663: session id = 3be34dcec5ad4ec7860fe6145ac58163-633e8264ce454183b2e0c0ad92f34629
I20240503 13:59:13.797139 2417 tablet_copy_source_session.cc:205] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:13.798131 4563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633e8264ce454183b2e0c0ad92f34629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.799929 4563 tablet_copy_client.cc:783] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:13.800092 4563 tablet_copy_client.cc:647] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:13.801299 4565 tablet_copy_client.cc:1002] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Injecting latency into file download: 1500
I20240503 13:59:13.827558 4575 ts_tablet_manager.cc:902] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac: Initiating tablet copy from peer f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:59:13.828066 4575 tablet_copy_client.cc:308] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44047
I20240503 13:59:13.831452 865 tablet_copy_service.cc:140] P f58fb372531d42a3bc405510bf5bced4: Received BeginTabletCopySession request for tablet 3cc710cdf7be47dcbb95ccc49e5beabf from peer ab254774096345abb0c1daf0f7afecac ({username='slave'} at 127.0.99.135:55613)
I20240503 13:59:13.831539 865 tablet_copy_service.cc:161] P f58fb372531d42a3bc405510bf5bced4: Beginning new tablet copy session on tablet 3cc710cdf7be47dcbb95ccc49e5beabf from peer ab254774096345abb0c1daf0f7afecac at {username='slave'} at 127.0.99.135:55613: session id = ab254774096345abb0c1daf0f7afecac-3cc710cdf7be47dcbb95ccc49e5beabf
I20240503 13:59:13.832228 865 tablet_copy_source_session.cc:205] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:13.832950 4575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc710cdf7be47dcbb95ccc49e5beabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.834785 4575 tablet_copy_client.cc:783] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:13.834945 4575 tablet_copy_client.cc:647] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:13.835911 4579 tablet_copy_client.cc:1002] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac: tablet copy: Injecting latency into file download: 1500
I20240503 13:59:13.948520 4583 ts_tablet_manager.cc:902] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc: Initiating tablet copy from peer e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971)
I20240503 13:59:13.949041 4583 tablet_copy_client.cc:308] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:40971
I20240503 13:59:13.949530 996 tablet_copy_service.cc:140] P e59e5d411bc046a28486330e3e4ffb66: Received BeginTabletCopySession request for tablet b0ecd805f87a4afeb6e41b1f6f671622 from peer 83070528d9574a528c6e57fd30b33dcc ({username='slave'} at 127.0.99.133:54493)
I20240503 13:59:13.949640 996 tablet_copy_service.cc:161] P e59e5d411bc046a28486330e3e4ffb66: Beginning new tablet copy session on tablet b0ecd805f87a4afeb6e41b1f6f671622 from peer 83070528d9574a528c6e57fd30b33dcc at {username='slave'} at 127.0.99.133:54493: session id = 83070528d9574a528c6e57fd30b33dcc-b0ecd805f87a4afeb6e41b1f6f671622
I20240503 13:59:13.950417 996 tablet_copy_source_session.cc:205] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:13.951294 4583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ecd805f87a4afeb6e41b1f6f671622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:13.953075 4583 tablet_copy_client.cc:783] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:13.953234 4583 tablet_copy_client.cc:647] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:13.954216 4587 tablet_copy_client.cc:1002] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Injecting latency into file download: 1500
W20240503 13:59:14.442767 425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240503 13:59:14.443614 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 633e8264ce454183b2e0c0ad92f34629: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:59:14.585513 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:59:14.831578 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 13:59:15.077363 909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 13:59:15.134967 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:59:15.301661 4563 tablet_copy_client.cc:523] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:15.303086 4563 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap starting.
W20240503 13:59:15.309448 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:59:15.329169 4563 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:15.329471 4563 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap complete.
I20240503 13:59:15.329646 4563 ts_tablet_manager.cc:1367] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163: Time spent bootstrapping tablet: real 0.027s user 0.024s sys 0.001s
I20240503 13:59:15.330067 4563 raft_consensus.cc:348] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
I20240503 13:59:15.330183 4563 raft_consensus.cc:729] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3be34dcec5ad4ec7860fe6145ac58163, State: Initialized, Role: LEARNER
I20240503 13:59:15.330281 4563 consensus_queue.cc:260] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
I20240503 13:59:15.330621 4563 ts_tablet_manager.cc:1398] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 13:59:15.331223 2417 tablet_copy_service.cc:342] P 06de56aefa784b4c8b7d5a3e7c5f9825: Request end of tablet copy session 3be34dcec5ad4ec7860fe6145ac58163-633e8264ce454183b2e0c0ad92f34629 received from {username='slave'} at 127.0.99.134:40663
I20240503 13:59:15.331373 2417 tablet_copy_service.cc:433] P 06de56aefa784b4c8b7d5a3e7c5f9825: ending tablet copy session 3be34dcec5ad4ec7860fe6145ac58163-633e8264ce454183b2e0c0ad92f34629 on tablet 633e8264ce454183b2e0c0ad92f34629 with peer 3be34dcec5ad4ec7860fe6145ac58163
I20240503 13:59:15.336241 4575 tablet_copy_client.cc:523] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:15.337698 4575 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac: Bootstrap starting.
I20240503 13:59:15.355581 4575 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:15.355885 4575 tablet_bootstrap.cc:492] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac: Bootstrap complete.
I20240503 13:59:15.356074 4575 ts_tablet_manager.cc:1367] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac: Time spent bootstrapping tablet: real 0.018s user 0.019s sys 0.000s
I20240503 13:59:15.356473 4575 raft_consensus.cc:348] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:59:15.356586 4575 raft_consensus.cc:729] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab254774096345abb0c1daf0f7afecac, State: Initialized, Role: LEARNER
I20240503 13:59:15.356689 4575 consensus_queue.cc:260] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:59:15.357012 4575 ts_tablet_manager.cc:1398] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:15.357579 865 tablet_copy_service.cc:342] P f58fb372531d42a3bc405510bf5bced4: Request end of tablet copy session ab254774096345abb0c1daf0f7afecac-3cc710cdf7be47dcbb95ccc49e5beabf received from {username='slave'} at 127.0.99.135:55613
I20240503 13:59:15.357677 865 tablet_copy_service.cc:433] P f58fb372531d42a3bc405510bf5bced4: ending tablet copy session ab254774096345abb0c1daf0f7afecac-3cc710cdf7be47dcbb95ccc49e5beabf on tablet 3cc710cdf7be47dcbb95ccc49e5beabf with peer ab254774096345abb0c1daf0f7afecac
W20240503 13:59:15.368937 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:59:15.442960 2135 raft_consensus.cc:1203] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240503 13:59:15.454491 4583 tablet_copy_client.cc:523] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:15.455786 4583 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc: Bootstrap starting.
I20240503 13:59:15.474557 4583 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:15.474875 4583 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc: Bootstrap complete.
I20240503 13:59:15.475071 4583 ts_tablet_manager.cc:1367] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.003s
I20240503 13:59:15.475514 4583 raft_consensus.cc:348] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:15.475625 4583 raft_consensus.cc:729] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83070528d9574a528c6e57fd30b33dcc, State: Initialized, Role: LEARNER
I20240503 13:59:15.475721 4583 consensus_queue.cc:260] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: true } }
I20240503 13:59:15.476022 4583 ts_tablet_manager.cc:1398] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 13:59:15.476620 996 tablet_copy_service.cc:342] P e59e5d411bc046a28486330e3e4ffb66: Request end of tablet copy session 83070528d9574a528c6e57fd30b33dcc-b0ecd805f87a4afeb6e41b1f6f671622 received from {username='slave'} at 127.0.99.133:54493
I20240503 13:59:15.476696 996 tablet_copy_service.cc:433] P e59e5d411bc046a28486330e3e4ffb66: ending tablet copy session 83070528d9574a528c6e57fd30b33dcc-b0ecd805f87a4afeb6e41b1f6f671622 on tablet b0ecd805f87a4afeb6e41b1f6f671622 with peer 83070528d9574a528c6e57fd30b33dcc
I20240503 13:59:15.688489 2004 raft_consensus.cc:1203] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240503 13:59:15.827353 647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 13:59:15.831228 2266 raft_consensus.cc:1203] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240503 13:59:15.973980 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:59:16.064699 4761 raft_consensus.cc:1050] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825: attempting to promote NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 to VOTER
I20240503 13:59:16.065141 4761 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } }
I20240503 13:59:16.066574 2135 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 13:59:16.066620 714 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 13:59:16.066677 2004 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 13:59:16.067090 4763 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 13:59:16.067360 4761 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 13:59:16.067556 4660 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 13:59:16.069401 4660 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:16.069763 713 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:16.071074 2004 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:16.072017 2135 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:16.072146 437 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: config changed from index 12 to 13, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:16.076869 2397 consensus_queue.cc:237] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } }
I20240503 13:59:16.077845 2135 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 FOLLOWER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 13:59:16.077885 2004 raft_consensus.cc:1261] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer 06de56aefa784b4c8b7d5a3e7c5f9825: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 13:59:16.078191 4760 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 13:59:16.078361 4660 consensus_queue.cc:1035] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 13:59:16.079949 4660 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:16.080312 2135 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:16.081256 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 633e8264ce454183b2e0c0ad92f34629 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:16.082394 441 catalog_manager.cc:5484] T 633e8264ce454183b2e0c0ad92f34629 P 06de56aefa784b4c8b7d5a3e7c5f9825 reported cstate change: config changed from index 13 to 14, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:16.083014 2004 raft_consensus.cc:2932] T 633e8264ce454183b2e0c0ad92f34629 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:16.084631 693 tablet_service.cc:1514] Processing DeleteTablet for tablet 633e8264ce454183b2e0c0ad92f34629 with delete_type TABLET_DATA_TOMBSTONED (TS 37dbd0b8eaae46d59b318fbbb4bab576 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:39974
I20240503 13:59:16.084996 4772 tablet_replica.cc:330] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576: stopping tablet replica
I20240503 13:59:16.085127 4772 raft_consensus.cc:2229] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:16.085359 4772 raft_consensus.cc:2258] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:16.086109 4772 ts_tablet_manager.cc:1875] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:16.087793 4772 ts_tablet_manager.cc:1888] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240503 13:59:16.087855 4772 log.cc:1196] T 633e8264ce454183b2e0c0ad92f34629 P 37dbd0b8eaae46d59b318fbbb4bab576: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/633e8264ce454183b2e0c0ad92f34629
I20240503 13:59:16.088303 428 catalog_manager.cc:4830] TS 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209): tablet 633e8264ce454183b2e0c0ad92f34629 (table rebalance_test_table_0 [id=6fb6addecd294243a4c390358564540d]) successfully deleted
W20240503 13:59:16.156721 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 13:59:16.241781 4799 raft_consensus.cc:1050] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: attempting to promote NON_VOTER ab254774096345abb0c1daf0f7afecac to VOTER
I20240503 13:59:16.242174 4799 consensus_queue.cc:237] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:16.243610 2266 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac [term 1 LEARNER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:16.243664 976 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:16.243971 2004 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:16.244104 4799 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:16.244346 4801 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:16.244498 4799 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:16.246160 4801 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:16.246578 2266 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:16.247006 976 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:16.248790 2004 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:16.249156 441 catalog_manager.cc:5484] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index 7 to 8, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:16.256248 4805 raft_consensus.cc:1050] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: attempting to promote NON_VOTER 83070528d9574a528c6e57fd30b33dcc to VOTER
I20240503 13:59:16.256706 4805 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:16.257865 2004 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 1 LEARNER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:16.258024 2397 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:16.258082 713 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Refusing update from remote peer e59e5d411bc046a28486330e3e4ffb66: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 13:59:16.258271 4744 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:16.258829 4744 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:16.258983 4511 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 13:59:16.261436 4511 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:16.261931 713 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:16.263325 2397 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:16.263474 2004 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:16.265674 434 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc reported cstate change: config changed from index 7 to 8, 83070528d9574a528c6e57fd30b33dcc (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e59e5d411bc046a28486330e3e4ffb66" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:16.416323 843 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3cc710cdf7be47dcbb95ccc49e5beabf"
dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34650
I20240503 13:59:16.416518 843 raft_consensus.cc:595] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:16.425984 976 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622"
dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34254
I20240503 13:59:16.426110 976 raft_consensus.cc:595] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:16.625041 4799 raft_consensus.cc:979] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: : Instructing follower 83070528d9574a528c6e57fd30b33dcc to start an election
I20240503 13:59:16.625170 4799 raft_consensus.cc:1067] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Signalling peer 83070528d9574a528c6e57fd30b33dcc to start an election
I20240503 13:59:16.625711 2004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3cc710cdf7be47dcbb95ccc49e5beabf"
dest_uuid: "83070528d9574a528c6e57fd30b33dcc"
from {username='slave'} at 127.0.99.131:58881
I20240503 13:59:16.625900 2004 raft_consensus.cc:482] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:16.625974 2004 raft_consensus.cc:3037] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:16.626889 2004 raft_consensus.cc:504] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:16.628245 2004 leader_election.cc:288] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [CANDIDATE]: Term 2 election: Requested vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971), ab254774096345abb0c1daf0f7afecac (127.0.99.135:41905)
I20240503 13:59:16.628636 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc710cdf7be47dcbb95ccc49e5beabf" candidate_uuid: "83070528d9574a528c6e57fd30b33dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:59:16.628783 843 raft_consensus.cc:3032] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:16.628839 843 raft_consensus.cc:729] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Running, Role: LEADER
I20240503 13:59:16.629007 843 consensus_queue.cc:260] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:16.629210 843 raft_consensus.cc:3037] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:16.629514 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc710cdf7be47dcbb95ccc49e5beabf" candidate_uuid: "83070528d9574a528c6e57fd30b33dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
I20240503 13:59:16.629653 976 raft_consensus.cc:3037] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:16.630335 843 raft_consensus.cc:2454] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83070528d9574a528c6e57fd30b33dcc in term 2.
I20240503 13:59:16.630574 976 raft_consensus.cc:2454] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83070528d9574a528c6e57fd30b33dcc in term 2.
I20240503 13:59:16.631052 1937 leader_election.cc:302] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 83070528d9574a528c6e57fd30b33dcc, e59e5d411bc046a28486330e3e4ffb66, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:59:16.631309 4713 raft_consensus.cc:2781] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:16.631510 4713 raft_consensus.cc:686] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 2 LEADER]: Becoming Leader. State: Replica: 83070528d9574a528c6e57fd30b33dcc, State: Running, Role: LEADER
I20240503 13:59:16.631800 4713 consensus_queue.cc:237] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:16.634270 2266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cc710cdf7be47dcbb95ccc49e5beabf" candidate_uuid: "83070528d9574a528c6e57fd30b33dcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ab254774096345abb0c1daf0f7afecac"
I20240503 13:59:16.634394 2266 raft_consensus.cc:3037] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:16.634516 441 catalog_manager.cc:5484] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc reported cstate change: term changed from 1 to 2, leader changed from f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) to 83070528d9574a528c6e57fd30b33dcc (127.0.99.133). New cstate: current_term: 2 leader_uuid: "83070528d9574a528c6e57fd30b33dcc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:16.635661 2266 raft_consensus.cc:2454] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83070528d9574a528c6e57fd30b33dcc in term 2.
I20240503 13:59:16.646276 976 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622"
dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34278
I20240503 13:59:16.646425 976 raft_consensus.cc:595] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:16.731834 4807 raft_consensus.cc:979] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: : Instructing follower 37dbd0b8eaae46d59b318fbbb4bab576 to start an election
I20240503 13:59:16.731956 4807 raft_consensus.cc:1067] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Signalling peer 37dbd0b8eaae46d59b318fbbb4bab576 to start an election
I20240503 13:59:16.732391 713 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622"
dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
from {username='slave'} at 127.0.99.132:50435
I20240503 13:59:16.732559 713 raft_consensus.cc:482] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:16.732633 713 raft_consensus.cc:3037] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:16.733793 713 raft_consensus.cc:504] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:16.734488 713 leader_election.cc:288] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 2 election: Requested vote from peers e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971), 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136:33227), 83070528d9574a528c6e57fd30b33dcc (127.0.99.133:34511)
I20240503 13:59:16.734913 2004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "83070528d9574a528c6e57fd30b33dcc"
I20240503 13:59:16.734853 2397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825"
I20240503 13:59:16.735056 2004 raft_consensus.cc:3037] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:16.735602 2397 raft_consensus.cc:3037] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:16.736158 2004 raft_consensus.cc:2454] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 2.
I20240503 13:59:16.736640 2397 raft_consensus.cc:2454] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 2.
I20240503 13:59:16.736996 650 leader_election.cc:302] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 06de56aefa784b4c8b7d5a3e7c5f9825, 37dbd0b8eaae46d59b318fbbb4bab576, 83070528d9574a528c6e57fd30b33dcc; no voters:
I20240503 13:59:16.737090 976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622" candidate_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e59e5d411bc046a28486330e3e4ffb66"
I20240503 13:59:16.737210 976 raft_consensus.cc:3032] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:16.737250 976 raft_consensus.cc:729] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e59e5d411bc046a28486330e3e4ffb66, State: Running, Role: LEADER
I20240503 13:59:16.737396 976 consensus_queue.cc:260] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:16.737571 976 raft_consensus.cc:3037] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:16.738276 976 raft_consensus.cc:2454] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37dbd0b8eaae46d59b318fbbb4bab576 in term 2.
I20240503 13:59:16.738453 4435 raft_consensus.cc:2781] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:16.738626 4435 raft_consensus.cc:686] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Becoming Leader. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Running, Role: LEADER
I20240503 13:59:16.738780 4435 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:16.740453 441 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: term changed from 1 to 2, leader changed from e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132) to 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:17.086858 976 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Refusing update from remote peer 83070528d9574a528c6e57fd30b33dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 13:59:17.087329 4713 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 13:59:17.117767 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 13:59:17.165931 2266 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Refusing update from remote peer 83070528d9574a528c6e57fd30b33dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 13:59:17.166428 4713 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:17.169895 843 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Refusing update from remote peer 83070528d9574a528c6e57fd30b33dcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 13:59:17.170411 4873 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:17.178421 2004 consensus_queue.cc:237] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:17.179642 2266 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Refusing update from remote peer 83070528d9574a528c6e57fd30b33dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:17.179668 976 raft_consensus.cc:1261] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Refusing update from remote peer 83070528d9574a528c6e57fd30b33dcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:17.180055 4877 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:17.180189 4873 consensus_queue.cc:1035] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:17.182209 4875 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:17.182716 976 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:17.184268 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3cc710cdf7be47dcbb95ccc49e5beabf with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:17.184551 441 catalog_manager.cc:5484] T 3cc710cdf7be47dcbb95ccc49e5beabf P 83070528d9574a528c6e57fd30b33dcc reported cstate change: config changed from index 8 to 10, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "83070528d9574a528c6e57fd30b33dcc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:17.184684 2266 raft_consensus.cc:2932] T 3cc710cdf7be47dcbb95ccc49e5beabf P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:17.187537 824 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cc710cdf7be47dcbb95ccc49e5beabf with delete_type TABLET_DATA_TOMBSTONED (TS f58fb372531d42a3bc405510bf5bced4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45588
I20240503 13:59:17.188102 4881 tablet_replica.cc:330] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: stopping tablet replica
I20240503 13:59:17.188423 4881 raft_consensus.cc:2229] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:17.188555 4881 raft_consensus.cc:2258] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:17.189415 4881 ts_tablet_manager.cc:1875] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:17.189918 976 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 13:59:17.190495 4435 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:17.191921 4881 ts_tablet_manager.cc:1888] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 13:59:17.192037 4881 log.cc:1196] T 3cc710cdf7be47dcbb95ccc49e5beabf P f58fb372531d42a3bc405510bf5bced4: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3cc710cdf7be47dcbb95ccc49e5beabf
I20240503 13:59:17.192608 427 catalog_manager.cc:4830] TS f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047): tablet 3cc710cdf7be47dcbb95ccc49e5beabf (table rebalance_test_table_1 [id=3d1bdf3fadac4bdaad04db8e072c3f98]) successfully deleted
I20240503 13:59:17.292099 843 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
I20240503 13:59:17.293776 2266 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 13:59:17.294166 2004 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 13:59:17.295418 4899 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 13:59:17.295719 779 consensus_peers.cc:485] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 -> Peer 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134:36679): Couldn't send request to peer 3be34dcec5ad4ec7860fe6145ac58163. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e521d523c904d659b745874eec536e4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:17.295864 4799 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:17.297668 4899 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } } }
I20240503 13:59:17.298293 2004 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } } }
I20240503 13:59:17.300797 2266 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } } }
I20240503 13:59:17.300864 432 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index 11 to 12, NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:17.326120 2397 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 13:59:17.326584 4435 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 13:59:17.332435 2004 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 13:59:17.333305 4916 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 13:59:17.583086 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:59:17.587069 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eb3d68a0d39042c98d02c9a93307bb88: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:59:17.704358 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:59:17.748950 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 13:59:17.755026 4943 ts_tablet_manager.cc:902] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163: Initiating tablet copy from peer f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047)
I20240503 13:59:17.755573 4943 tablet_copy_client.cc:308] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:44047
I20240503 13:59:17.756086 865 tablet_copy_service.cc:140] P f58fb372531d42a3bc405510bf5bced4: Received BeginTabletCopySession request for tablet 6e521d523c904d659b745874eec536e4 from peer 3be34dcec5ad4ec7860fe6145ac58163 ({username='slave'} at 127.0.99.134:35791)
I20240503 13:59:17.756203 865 tablet_copy_service.cc:161] P f58fb372531d42a3bc405510bf5bced4: Beginning new tablet copy session on tablet 6e521d523c904d659b745874eec536e4 from peer 3be34dcec5ad4ec7860fe6145ac58163 at {username='slave'} at 127.0.99.134:35791: session id = 3be34dcec5ad4ec7860fe6145ac58163-6e521d523c904d659b745874eec536e4
I20240503 13:59:17.756798 713 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } }
I20240503 13:59:17.757001 865 tablet_copy_source_session.cc:205] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:17.757987 4943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e521d523c904d659b745874eec536e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:17.758023 2004 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:17.758128 2397 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 13:59:17.758553 4916 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:17.758922 4916 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 13:59:17.760411 4943 tablet_copy_client.cc:783] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:17.760601 4943 tablet_copy_client.cc:647] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:17.760538 4914 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:17.761061 2004 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:17.761291 2397 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } }
I20240503 13:59:17.761698 4947 tablet_copy_client.cc:1002] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Injecting latency into file download: 1500
I20240503 13:59:17.762233 428 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0ecd805f87a4afeb6e41b1f6f671622 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:17.763429 441 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 reported cstate change: config changed from index 8 to 10, VOTER e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:17.765772 956 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ecd805f87a4afeb6e41b1f6f671622 with delete_type TABLET_DATA_TOMBSTONED (TS e59e5d411bc046a28486330e3e4ffb66 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41916
I20240503 13:59:17.766134 4948 tablet_replica.cc:330] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: stopping tablet replica
I20240503 13:59:17.766436 4948 raft_consensus.cc:2229] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:17.766628 4948 raft_consensus.cc:2258] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:17.767362 4948 ts_tablet_manager.cc:1875] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:17.769121 4948 ts_tablet_manager.cc:1888] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 13:59:17.769184 4948 log.cc:1196] T b0ecd805f87a4afeb6e41b1f6f671622 P e59e5d411bc046a28486330e3e4ffb66: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b0ecd805f87a4afeb6e41b1f6f671622
I20240503 13:59:17.769655 425 catalog_manager.cc:4830] TS e59e5d411bc046a28486330e3e4ffb66 (127.0.99.132:40971): tablet b0ecd805f87a4afeb6e41b1f6f671622 (table rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]) successfully deleted
W20240503 13:59:17.995754 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:59:18.373725 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 13:59:18.773912 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 13:59:19.262017 4943 tablet_copy_client.cc:523] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:19.263746 4943 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap starting.
I20240503 13:59:19.283522 4943 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:19.283815 4943 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163: Bootstrap complete.
I20240503 13:59:19.283982 4943 ts_tablet_manager.cc:1367] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240503 13:59:19.284396 4943 raft_consensus.cc:348] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
I20240503 13:59:19.284507 4943 raft_consensus.cc:729] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3be34dcec5ad4ec7860fe6145ac58163, State: Initialized, Role: LEARNER
I20240503 13:59:19.284590 4943 consensus_queue.cc:260] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: true } }
I20240503 13:59:19.284955 4943 ts_tablet_manager.cc:1398] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:19.285595 865 tablet_copy_service.cc:342] P f58fb372531d42a3bc405510bf5bced4: Request end of tablet copy session 3be34dcec5ad4ec7860fe6145ac58163-6e521d523c904d659b745874eec536e4 received from {username='slave'} at 127.0.99.134:35791
I20240503 13:59:19.285692 865 tablet_copy_service.cc:433] P f58fb372531d42a3bc405510bf5bced4: ending tablet copy session 3be34dcec5ad4ec7860fe6145ac58163-6e521d523c904d659b745874eec536e4 on tablet 6e521d523c904d659b745874eec536e4 with peer 3be34dcec5ad4ec7860fe6145ac58163
I20240503 13:59:19.488910 4919 consensus_queue.cc:579] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Leader has been unable to successfully communicate with peer 769439c9b5d3418eb0ebd0287954e6cc for more than 30 seconds (30.286s)
I20240503 13:59:19.564639 2135 raft_consensus.cc:1203] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240503 13:59:19.738687 5011 consensus_queue.cc:579] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Leader has been unable to successfully communicate with peer 769439c9b5d3418eb0ebd0287954e6cc for more than 30 seconds (30.097s)
I20240503 13:59:19.806591 4919 consensus_queue.cc:579] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Leader has been unable to successfully communicate with peer 769439c9b5d3418eb0ebd0287954e6cc for more than 30 seconds (30.536s)
W20240503 13:59:19.808038 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:59:19.902348 4916 consensus_queue.cc:579] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Leader has been unable to successfully communicate with peer 769439c9b5d3418eb0ebd0287954e6cc for more than 30 seconds (30.588s)
I20240503 13:59:19.949669 5028 raft_consensus.cc:1050] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: attempting to promote NON_VOTER 3be34dcec5ad4ec7860fe6145ac58163 to VOTER
I20240503 13:59:19.950098 5028 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } }
I20240503 13:59:19.951519 2135 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 LEARNER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 13:59:19.951628 2266 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 13:59:19.951690 2004 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer f58fb372531d42a3bc405510bf5bced4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 13:59:19.952041 5027 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 13:59:19.952243 5030 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 13:59:19.952571 5028 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 13:59:19.954320 5027 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:19.955255 2004 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:19.955799 2266 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:19.957264 432 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 reported cstate change: config changed from index 12 to 13, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f58fb372531d42a3bc405510bf5bced4" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 13:59:19.957782 2135 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:20.033305 843 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e521d523c904d659b745874eec536e4"
dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58808
I20240503 13:59:20.033468 843 raft_consensus.cc:595] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 LEADER]: Received request to transfer leadership
I20240503 13:59:20.051112 4919 consensus_queue.cc:579] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Leader has been unable to successfully communicate with peer 769439c9b5d3418eb0ebd0287954e6cc for more than 30 seconds (30.395s)
W20240503 13:59:20.069113 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:59:20.138808 909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 13:59:20.139508 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:59:20.244647 843 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e521d523c904d659b745874eec536e4"
dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58832
I20240503 13:59:20.244781 843 raft_consensus.cc:595] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 LEADER]: Received request to transfer leadership
I20240503 13:59:20.418577 5030 raft_consensus.cc:979] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: : Instructing follower 3be34dcec5ad4ec7860fe6145ac58163 to start an election
I20240503 13:59:20.418718 5030 raft_consensus.cc:1067] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 LEADER]: Signalling peer 3be34dcec5ad4ec7860fe6145ac58163 to start an election
I20240503 13:59:20.419176 2135 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e521d523c904d659b745874eec536e4"
dest_uuid: "3be34dcec5ad4ec7860fe6145ac58163"
from {username='slave'} at 127.0.99.131:59865
I20240503 13:59:20.419389 2135 raft_consensus.cc:482] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:20.419478 2135 raft_consensus.cc:3037] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [term 2 FOLLOWER]: Advancing to term 3
I20240503 13:59:20.420338 2135 raft_consensus.cc:504] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } }
I20240503 13:59:20.421363 2135 leader_election.cc:288] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [CANDIDATE]: Term 3 election: Requested vote from peers f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047), 83070528d9574a528c6e57fd30b33dcc (127.0.99.133:34511), ab254774096345abb0c1daf0f7afecac (127.0.99.135:41905)
I20240503 13:59:20.421672 843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e521d523c904d659b745874eec536e4" candidate_uuid: "3be34dcec5ad4ec7860fe6145ac58163" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "f58fb372531d42a3bc405510bf5bced4"
I20240503 13:59:20.421806 843 raft_consensus.cc:3032] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 13:59:20.421854 843 raft_consensus.cc:729] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f58fb372531d42a3bc405510bf5bced4, State: Running, Role: LEADER
I20240503 13:59:20.422013 843 consensus_queue.cc:260] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } }
I20240503 13:59:20.422224 843 raft_consensus.cc:3037] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 2 FOLLOWER]: Advancing to term 3
I20240503 13:59:20.423223 843 raft_consensus.cc:2454] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3be34dcec5ad4ec7860fe6145ac58163 in term 3.
I20240503 13:59:20.425284 2004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e521d523c904d659b745874eec536e4" candidate_uuid: "3be34dcec5ad4ec7860fe6145ac58163" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "83070528d9574a528c6e57fd30b33dcc"
I20240503 13:59:20.425397 2004 raft_consensus.cc:3037] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Advancing to term 3
I20240503 13:59:20.425679 2266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e521d523c904d659b745874eec536e4" candidate_uuid: "3be34dcec5ad4ec7860fe6145ac58163" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "ab254774096345abb0c1daf0f7afecac"
I20240503 13:59:20.425804 2266 raft_consensus.cc:3037] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Advancing to term 3
I20240503 13:59:20.426451 2004 raft_consensus.cc:2454] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3be34dcec5ad4ec7860fe6145ac58163 in term 3.
I20240503 13:59:20.426788 2266 raft_consensus.cc:2454] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3be34dcec5ad4ec7860fe6145ac58163 in term 3.
I20240503 13:59:20.426801 2071 leader_election.cc:302] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3be34dcec5ad4ec7860fe6145ac58163, 83070528d9574a528c6e57fd30b33dcc, f58fb372531d42a3bc405510bf5bced4; no voters:
I20240503 13:59:20.427060 5003 raft_consensus.cc:2781] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 13:59:20.427282 5003 raft_consensus.cc:686] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [term 3 LEADER]: Becoming Leader. State: Replica: 3be34dcec5ad4ec7860fe6145ac58163, State: Running, Role: LEADER
I20240503 13:59:20.427616 5003 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } }
I20240503 13:59:20.430173 441 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 reported cstate change: term changed from 2 to 3, leader changed from f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) to 3be34dcec5ad4ec7860fe6145ac58163 (127.0.99.134). New cstate: current_term: 3 leader_uuid: "3be34dcec5ad4ec7860fe6145ac58163" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:59:20.514459 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:59:20.515812 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 13:59:20.580274 425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 13:59:20.581043 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6e521d523c904d659b745874eec536e4: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:59:20.843209 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:59:20.891359 843 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 3 FOLLOWER]: Refusing update from remote peer 3be34dcec5ad4ec7860fe6145ac58163: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 13:59:20.891928 5003 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 13:59:20.950788 2004 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 3 FOLLOWER]: Refusing update from remote peer 3be34dcec5ad4ec7860fe6145ac58163: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 13:59:20.951305 5003 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 13:59:20.954195 2266 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 3 FOLLOWER]: Refusing update from remote peer 3be34dcec5ad4ec7860fe6145ac58163: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 13:59:20.955271 5003 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 13:59:21.012135 647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 13:59:21.150173 4919 consensus_queue.cc:579] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Leader has been unable to successfully communicate with peer 769439c9b5d3418eb0ebd0287954e6cc for more than 30 seconds (30.068s)
W20240503 13:59:21.151695 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 13:59:21.333791 5098 consensus_queue.cc:579] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 [LEADER]: Leader has been unable to successfully communicate with peer 769439c9b5d3418eb0ebd0287954e6cc for more than 30 seconds (30.274s)
I20240503 13:59:21.400132 2135 consensus_queue.cc:237] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } }
I20240503 13:59:21.401355 2004 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 3 FOLLOWER]: Refusing update from remote peer 3be34dcec5ad4ec7860fe6145ac58163: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240503 13:59:21.401355 2266 raft_consensus.cc:1261] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 3 FOLLOWER]: Refusing update from remote peer 3be34dcec5ad4ec7860fe6145ac58163: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240503 13:59:21.401768 5085 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 13:59:21.402004 5087 consensus_queue.cc:1035] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 13:59:21.403614 5003 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 3be34dcec5ad4ec7860fe6145ac58163 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:21.404042 2004 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P 83070528d9574a528c6e57fd30b33dcc [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:21.404119 2266 raft_consensus.cc:2932] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:21.406152 426 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e521d523c904d659b745874eec536e4 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:21.406949 437 catalog_manager.cc:5484] T 6e521d523c904d659b745874eec536e4 P ab254774096345abb0c1daf0f7afecac reported cstate change: config changed from index 13 to 15, VOTER f58fb372531d42a3bc405510bf5bced4 (127.0.99.131) evicted. New cstate: current_term: 3 leader_uuid: "3be34dcec5ad4ec7860fe6145ac58163" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } peers { permanent_uuid: "3be34dcec5ad4ec7860fe6145ac58163" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 36679 } attrs { promote: false } } }
I20240503 13:59:21.409824 824 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e521d523c904d659b745874eec536e4 with delete_type TABLET_DATA_TOMBSTONED (TS f58fb372531d42a3bc405510bf5bced4 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:45588
I20240503 13:59:21.410215 5108 tablet_replica.cc:330] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: stopping tablet replica
I20240503 13:59:21.410368 5108 raft_consensus.cc:2229] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:21.410612 5108 raft_consensus.cc:2258] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:21.411291 5108 ts_tablet_manager.cc:1875] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:21.413287 5108 ts_tablet_manager.cc:1888] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20240503 13:59:21.413353 5108 log.cc:1196] T 6e521d523c904d659b745874eec536e4 P f58fb372531d42a3bc405510bf5bced4: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6e521d523c904d659b745874eec536e4
I20240503 13:59:21.413885 427 catalog_manager.cc:4830] TS f58fb372531d42a3bc405510bf5bced4 (127.0.99.131:44047): tablet 6e521d523c904d659b745874eec536e4 (table rebalance_test_table_4 [id=d28850114dae4436ae20500518f1e627]) successfully deleted
I20240503 13:59:21.538789 2377 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:21.544541 2246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:21.544975 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:21.546931 2115 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:21.548342 693 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:21.550225 1984 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:21.561059 824 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:21.600144 713 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:59:21.601593 2004 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 13:59:21.601646 2397 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
W20240503 13:59:21.601681 648 consensus_peers.cc:485] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer ab254774096345abb0c1daf0f7afecac (127.0.99.135:41905): Couldn't send request to peer ab254774096345abb0c1daf0f7afecac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0ecd805f87a4afeb6e41b1f6f671622. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:21.602026 5160 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:21.602229 5159 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 13:59:21.604043 5079 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:59:21.605701 2004 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:59:21.605749 2397 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
I20240503 13:59:21.608134 434 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 reported cstate change: config changed from index 10 to 11, NON_VOTER ab254774096345abb0c1daf0f7afecac (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } } }
W20240503 13:59:22.139911 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 13:59:22.212127 5182 ts_tablet_manager.cc:902] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac: Initiating tablet copy from peer 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209)
I20240503 13:59:22.212706 5182 tablet_copy_client.cc:308] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:42209
I20240503 13:59:22.213207 734 tablet_copy_service.cc:140] P 37dbd0b8eaae46d59b318fbbb4bab576: Received BeginTabletCopySession request for tablet b0ecd805f87a4afeb6e41b1f6f671622 from peer ab254774096345abb0c1daf0f7afecac ({username='slave'} at 127.0.99.135:57933)
I20240503 13:59:22.213317 734 tablet_copy_service.cc:161] P 37dbd0b8eaae46d59b318fbbb4bab576: Beginning new tablet copy session on tablet b0ecd805f87a4afeb6e41b1f6f671622 from peer ab254774096345abb0c1daf0f7afecac at {username='slave'} at 127.0.99.135:57933: session id = ab254774096345abb0c1daf0f7afecac-b0ecd805f87a4afeb6e41b1f6f671622
I20240503 13:59:22.214200 734 tablet_copy_source_session.cc:205] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:22.215095 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ecd805f87a4afeb6e41b1f6f671622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:22.216964 5182 tablet_copy_client.cc:783] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:22.217133 5182 tablet_copy_client.cc:647] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:22.218150 5186 tablet_copy_client.cc:1002] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac: tablet copy: Injecting latency into file download: 1500
W20240503 13:59:22.575425 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:59:22.639715 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:59:22.829087 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:59:23.174856 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:59:23.379269 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 13:59:23.578791 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 13:59:23.718577 5182 tablet_copy_client.cc:523] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:23.720085 5182 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac: Bootstrap starting.
I20240503 13:59:23.740207 5182 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:23.740653 5182 tablet_bootstrap.cc:492] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac: Bootstrap complete.
I20240503 13:59:23.740866 5182 ts_tablet_manager.cc:1367] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.005s
I20240503 13:59:23.741277 5182 raft_consensus.cc:348] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:59:23.741425 5182 raft_consensus.cc:729] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ab254774096345abb0c1daf0f7afecac, State: Initialized, Role: LEARNER
I20240503 13:59:23.741550 5182 consensus_queue.cc:260] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:59:23.741892 5182 ts_tablet_manager.cc:1398] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:23.742477 734 tablet_copy_service.cc:342] P 37dbd0b8eaae46d59b318fbbb4bab576: Request end of tablet copy session ab254774096345abb0c1daf0f7afecac-b0ecd805f87a4afeb6e41b1f6f671622 received from {username='slave'} at 127.0.99.135:57933
I20240503 13:59:23.742589 734 tablet_copy_service.cc:433] P 37dbd0b8eaae46d59b318fbbb4bab576: ending tablet copy session ab254774096345abb0c1daf0f7afecac-b0ecd805f87a4afeb6e41b1f6f671622 on tablet b0ecd805f87a4afeb6e41b1f6f671622 with peer ab254774096345abb0c1daf0f7afecac
I20240503 13:59:23.786597 2266 raft_consensus.cc:1203] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20240503 13:59:24.192386 5252 raft_consensus.cc:1050] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: attempting to promote NON_VOTER ab254774096345abb0c1daf0f7afecac to VOTER
I20240503 13:59:24.192904 5252 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:24.194324 2266 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [term 2 LEARNER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 13:59:24.194378 2004 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 13:59:24.194610 2397 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 FOLLOWER]: Refusing update from remote peer 37dbd0b8eaae46d59b318fbbb4bab576: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 13:59:24.194751 5252 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:24.194957 5254 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:24.195183 5251 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 13:59:24.197214 5251 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:24.197669 2266 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:24.197757 2397 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:24.199719 2004 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:24.200325 432 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac reported cstate change: config changed from index 11 to 12, ab254774096345abb0c1daf0f7afecac (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:24.310818 713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622"
dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35946
I20240503 13:59:24.310953 713 raft_consensus.cc:595] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Received request to transfer leadership
I20240503 13:59:24.527163 713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622"
dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35970
I20240503 13:59:24.527331 713 raft_consensus.cc:595] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Received request to transfer leadership
I20240503 13:59:24.575650 5254 raft_consensus.cc:979] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: : Instructing follower ab254774096345abb0c1daf0f7afecac to start an election
I20240503 13:59:24.575799 5254 raft_consensus.cc:1067] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Signalling peer ab254774096345abb0c1daf0f7afecac to start an election
I20240503 13:59:24.576288 2266 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622"
dest_uuid: "ab254774096345abb0c1daf0f7afecac"
from {username='slave'} at 127.0.99.130:36123
I20240503 13:59:24.576467 2266 raft_consensus.cc:482] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:24.576520 2266 raft_consensus.cc:3037] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [term 2 FOLLOWER]: Advancing to term 3
I20240503 13:59:24.577392 2266 raft_consensus.cc:504] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:24.578408 2266 leader_election.cc:288] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [CANDIDATE]: Term 3 election: Requested vote from peers 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209), 06de56aefa784b4c8b7d5a3e7c5f9825 (127.0.99.136:33227), 83070528d9574a528c6e57fd30b33dcc (127.0.99.133:34511)
I20240503 13:59:24.578723 713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622" candidate_uuid: "ab254774096345abb0c1daf0f7afecac" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "37dbd0b8eaae46d59b318fbbb4bab576"
I20240503 13:59:24.578867 713 raft_consensus.cc:3032] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 13:59:24.578923 713 raft_consensus.cc:729] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 37dbd0b8eaae46d59b318fbbb4bab576, State: Running, Role: LEADER
I20240503 13:59:24.579104 713 consensus_queue.cc:260] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:24.579346 713 raft_consensus.cc:3037] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 2 FOLLOWER]: Advancing to term 3
I20240503 13:59:24.580435 713 raft_consensus.cc:2454] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab254774096345abb0c1daf0f7afecac in term 3.
I20240503 13:59:24.582549 2004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622" candidate_uuid: "ab254774096345abb0c1daf0f7afecac" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "83070528d9574a528c6e57fd30b33dcc"
I20240503 13:59:24.582597 2397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ecd805f87a4afeb6e41b1f6f671622" candidate_uuid: "ab254774096345abb0c1daf0f7afecac" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825"
I20240503 13:59:24.582669 2004 raft_consensus.cc:3037] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 2 FOLLOWER]: Advancing to term 3
I20240503 13:59:24.582743 2397 raft_consensus.cc:3037] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 2 FOLLOWER]: Advancing to term 3
I20240503 13:59:24.583557 2397 raft_consensus.cc:2454] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab254774096345abb0c1daf0f7afecac in term 3.
I20240503 13:59:24.583557 2004 raft_consensus.cc:2454] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab254774096345abb0c1daf0f7afecac in term 3.
I20240503 13:59:24.583887 2202 leader_election.cc:302] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 06de56aefa784b4c8b7d5a3e7c5f9825, 37dbd0b8eaae46d59b318fbbb4bab576, ab254774096345abb0c1daf0f7afecac; no voters:
I20240503 13:59:24.584120 5240 raft_consensus.cc:2781] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [term 3 FOLLOWER]: Leader election won for term 3
I20240503 13:59:24.584324 5240 raft_consensus.cc:686] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [term 3 LEADER]: Becoming Leader. State: Replica: ab254774096345abb0c1daf0f7afecac, State: Running, Role: LEADER
I20240503 13:59:24.584692 5240 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:24.587114 441 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac reported cstate change: term changed from 2 to 3, leader changed from 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) to ab254774096345abb0c1daf0f7afecac (127.0.99.135). New cstate: current_term: 3 leader_uuid: "ab254774096345abb0c1daf0f7afecac" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 13:59:24.589979 425 catalog_manager.cc:4573] TS 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1d63d06c06ed4743b3e1176413142bef: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111)
W20240503 13:59:24.652531 909 consensus_peers.cc:485] T a67ac7eb49584e3e9a09b3b3f00785b1 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:59:25.089217 2397 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 3 FOLLOWER]: Refusing update from remote peer ab254774096345abb0c1daf0f7afecac: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 13:59:25.089723 5303 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 13:59:25.119451 713 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 3 FOLLOWER]: Refusing update from remote peer ab254774096345abb0c1daf0f7afecac: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 13:59:25.119920 5303 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 13:59:25.122694 2004 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 3 FOLLOWER]: Refusing update from remote peer ab254774096345abb0c1daf0f7afecac: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 13:59:25.123086 5303 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 13:59:25.146523 909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 13:59:25.147265 909 consensus_peers.cc:485] T 920e110f079f43dc89feb833979fdbc6 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:59:25.212095 909 consensus_peers.cc:485] T 857e4baba1984dc4af9c81ee2baf2fba P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:59:25.238058 909 consensus_peers.cc:485] T d8df290dcc75450b89cf9d3989ed7569 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:59:25.593168 647 consensus_peers.cc:485] T 8d06ae5dbfb44987b060c8740bfb231e P 37dbd0b8eaae46d59b318fbbb4bab576 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:59:25.595731 2266 consensus_queue.cc:237] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } }
I20240503 13:59:25.596886 2004 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 3 FOLLOWER]: Refusing update from remote peer ab254774096345abb0c1daf0f7afecac: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 13:59:25.597016 2397 raft_consensus.cc:1261] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 3 FOLLOWER]: Refusing update from remote peer ab254774096345abb0c1daf0f7afecac: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 13:59:25.597352 5306 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [LEADER]: Connected to new peer: Peer: permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 13:59:25.597659 5304 consensus_queue.cc:1035] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 13:59:25.599831 5303 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P ab254774096345abb0c1daf0f7afecac [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:25.600191 2397 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:25.601579 426 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0ecd805f87a4afeb6e41b1f6f671622 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 13:59:25.602803 437 catalog_manager.cc:5484] T b0ecd805f87a4afeb6e41b1f6f671622 P 06de56aefa784b4c8b7d5a3e7c5f9825 reported cstate change: config changed from index 12 to 14, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New cstate: current_term: 3 leader_uuid: "ab254774096345abb0c1daf0f7afecac" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:25.603621 2004 raft_consensus.cc:2932] T b0ecd805f87a4afeb6e41b1f6f671622 P 83070528d9574a528c6e57fd30b33dcc [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "06de56aefa784b4c8b7d5a3e7c5f9825" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 33227 } attrs { promote: false } } peers { permanent_uuid: "83070528d9574a528c6e57fd30b33dcc" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34511 } attrs { promote: false } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: false } } }
I20240503 13:59:25.607213 693 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ecd805f87a4afeb6e41b1f6f671622 with delete_type TABLET_DATA_TOMBSTONED (TS 37dbd0b8eaae46d59b318fbbb4bab576 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:39974
I20240503 13:59:25.607682 5333 tablet_replica.cc:330] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: stopping tablet replica
I20240503 13:59:25.607908 5333 raft_consensus.cc:2229] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:25.608063 5333 raft_consensus.cc:2258] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:25.608806 5333 ts_tablet_manager.cc:1875] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:25.610913 5333 ts_tablet_manager.cc:1888] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240503 13:59:25.610983 5333 log.cc:1196] T b0ecd805f87a4afeb6e41b1f6f671622 P 37dbd0b8eaae46d59b318fbbb4bab576: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b0ecd805f87a4afeb6e41b1f6f671622
I20240503 13:59:25.611660 428 catalog_manager.cc:4830] TS 37dbd0b8eaae46d59b318fbbb4bab576 (127.0.99.130:42209): tablet b0ecd805f87a4afeb6e41b1f6f671622 (table rebalance_test_table_3 [id=177f7ab486ff4ed8af087935d243db5d]) successfully deleted
I20240503 13:59:25.824782 1984 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.825634 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:25.826141 2115 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.832032 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:25.837955 2246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.847911 2377 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 13:59:25.855159 909 consensus_peers.cc:485] T 3fd93f79958e489b9028f5123c600f74 P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 13:59:25.857137 824 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.934401 2377 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.936887 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:25.939520 2246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.941244 2115 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.941304 824 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:25.949457 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:25.967586 1984 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:26.002028 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:33075
--local_ip_for_outbound_sockets=127.0.99.129
--tserver_master_addrs=127.0.99.190:40943
--webserver_port=37125
--webserver_interface=127.0.99.129
--builtin_ntp_servers=127.0.99.148:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 13:59:26.038816 909 consensus_peers.cc:485] T a35914f78ffe423e91ce866785764b9c P e59e5d411bc046a28486330e3e4ffb66 -> Peer 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075): Couldn't send request to peer 769439c9b5d3418eb0ebd0287954e6cc. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 13:59:26.111837 5410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:26.112046 5410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:26.112107 5410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:26.115177 5410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 13:59:26.115285 5410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:26.115399 5410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:59:26.118878 5410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:33075
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=37125
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.0.99.190:40943
--never_fsync=true
--heap_profile_path=/tmp/kudu.5410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:26.119748 5410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:26.120777 5410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:26.128643 647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:33075: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 13:59:26.129734 5416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.130410 5419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:26.131325 5410 server_base.cc:1034] running on GCE node
W20240503 13:59:26.132421 5417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:26.132814 5410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:26.133404 5410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:26.134562 5410 hybrid_clock.cc:648] HybridClock initialized: now 1714744766134526 us; error 41 us; skew 500 ppm
I20240503 13:59:26.136405 5410 webserver.cc:460] Webserver started at http://127.0.99.129:37125/ using document root <none> and password file <none>
I20240503 13:59:26.136960 5410 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:26.137042 5410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:26.140401 5410 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20240503 13:59:26.142984 5427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.143996 5410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:26.144097 5410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "769439c9b5d3418eb0ebd0287954e6cc"
format_stamp: "Formatted at 2024-05-03 13:58:38 on dist-test-slave-snql"
I20240503 13:59:26.144503 5410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:26.162748 5410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:26.163441 5410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:26.163621 5410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:26.164263 5410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:26.165445 5434 ts_tablet_manager.cc:516] Loading tablet metadata (0/12 complete)
I20240503 13:59:26.174856 5410 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240503 13:59:26.174923 5410 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20240503 13:59:26.174980 5410 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240503 13:59:26.177217 5434 tablet_bootstrap.cc:492] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.181061 5410 ts_tablet_manager.cc:590] Registered 12 tablets
I20240503 13:59:26.181109 5410 ts_tablet_manager.cc:569] Time spent register tablets: real 0.006s user 0.004s sys 0.000s
I20240503 13:59:26.191105 5410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33075
I20240503 13:59:26.191289 5541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33075 every 8 connection(s)
I20240503 13:59:26.192122 5434 log.cc:824] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:26.192283 5410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:59:26.198297 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 5410
I20240503 13:59:26.200615 5543 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40943
I20240503 13:59:26.200840 5543 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:26.201401 5543 heartbeater.cc:511] Master 127.0.99.190:40943 requested a full tablet report, sending...
I20240503 13:59:26.203145 438 ts_manager.cc:194] Re-registered known tserver with Master: 769439c9b5d3418eb0ebd0287954e6cc (127.0.99.129:33075)
I20240503 13:59:26.205104 438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:34933
I20240503 13:59:26.220769 5434 tablet_bootstrap.cc:492] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.221258 5434 tablet_bootstrap.cc:492] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.222609 5434 ts_tablet_manager.cc:1367] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.046s user 0.037s sys 0.004s
I20240503 13:59:26.225085 5434 raft_consensus.cc:348] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:59:26.225512 5434 raft_consensus.cc:729] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.226019 5434 consensus_queue.cc:260] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:59:26.226656 5543 heartbeater.cc:503] Master 127.0.99.190:40943 was elected leader, sending a full tablet report...
I20240503 13:59:26.226873 5434 ts_tablet_manager.cc:1398] T a67ac7eb49584e3e9a09b3b3f00785b1 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20240503 13:59:26.227187 5434 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.255061 5434 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.255419 5434 tablet_bootstrap.cc:492] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.256287 5434 ts_tablet_manager.cc:1367] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.029s user 0.028s sys 0.000s
I20240503 13:59:26.256755 5434 raft_consensus.cc:348] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:59:26.256865 5434 raft_consensus.cc:729] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.257000 5434 consensus_queue.cc:260] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:59:26.257236 5434 ts_tablet_manager.cc:1398] T 6e521d523c904d659b745874eec536e4 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:26.257427 5434 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.284713 5434 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.285056 5434 tablet_bootstrap.cc:492] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.285928 5434 ts_tablet_manager.cc:1367] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.004s
I20240503 13:59:26.286305 5434 raft_consensus.cc:348] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:59:26.286398 5434 raft_consensus.cc:729] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.286516 5434 consensus_queue.cc:260] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:59:26.286732 5434 ts_tablet_manager.cc:1398] T 67ee1c64f81b492a9c83e7618878ff79 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:26.286870 5434 tablet_bootstrap.cc:492] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.311847 5434 tablet_bootstrap.cc:492] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.312193 5434 tablet_bootstrap.cc:492] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.313146 5434 ts_tablet_manager.cc:1367] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.004s
I20240503 13:59:26.313513 5434 raft_consensus.cc:348] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:59:26.313609 5434 raft_consensus.cc:729] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.313751 5434 consensus_queue.cc:260] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:59:26.313985 5434 ts_tablet_manager.cc:1398] T 857e4baba1984dc4af9c81ee2baf2fba P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:26.314116 5434 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.336956 5434 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.337234 5434 tablet_bootstrap.cc:492] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.338064 5434 ts_tablet_manager.cc:1367] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.004s
I20240503 13:59:26.338415 5434 raft_consensus.cc:348] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:59:26.338510 5434 raft_consensus.cc:729] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.338655 5434 consensus_queue.cc:260] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } }
I20240503 13:59:26.338878 5434 ts_tablet_manager.cc:1398] T 633e8264ce454183b2e0c0ad92f34629 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:26.339016 5434 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.364317 5434 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.364632 5434 tablet_bootstrap.cc:492] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.365533 5434 ts_tablet_manager.cc:1367] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.008s
I20240503 13:59:26.365970 5434 raft_consensus.cc:348] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:59:26.366092 5434 raft_consensus.cc:729] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.366250 5434 consensus_queue.cc:260] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } attrs { replace: true } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "ab254774096345abb0c1daf0f7afecac" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 41905 } attrs { promote: true } }
I20240503 13:59:26.366533 5434 ts_tablet_manager.cc:1398] T 1d63d06c06ed4743b3e1176413142bef P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:26.366669 5434 tablet_bootstrap.cc:492] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.393942 5434 tablet_bootstrap.cc:492] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.394305 5434 tablet_bootstrap.cc:492] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.395462 5434 ts_tablet_manager.cc:1367] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.011s
I20240503 13:59:26.395864 5434 raft_consensus.cc:348] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:59:26.395970 5434 raft_consensus.cc:729] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.396080 5434 consensus_queue.cc:260] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:59:26.396301 5434 ts_tablet_manager.cc:1398] T 8d06ae5dbfb44987b060c8740bfb231e P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:26.396873 5434 tablet_bootstrap.cc:492] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.419317 2377 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:26.419638 2115 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:26.422693 5476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:26.423367 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:26.428279 5434 tablet_bootstrap.cc:492] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.428586 5434 tablet_bootstrap.cc:492] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.429735 5434 ts_tablet_manager.cc:1367] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.005s
I20240503 13:59:26.430109 5434 raft_consensus.cc:348] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:59:26.430202 5434 raft_consensus.cc:729] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.430331 5434 consensus_queue.cc:260] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:59:26.430567 5434 ts_tablet_manager.cc:1398] T 920e110f079f43dc89feb833979fdbc6 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 13:59:26.430680 5434 tablet_bootstrap.cc:492] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.449472 1984 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:26.458160 2246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:26.463773 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:26.463888 824 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:26.497143 5434 tablet_bootstrap.cc:492] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.497473 5434 tablet_bootstrap.cc:492] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.498322 5434 ts_tablet_manager.cc:1367] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.068s user 0.019s sys 0.004s
I20240503 13:59:26.498716 5434 raft_consensus.cc:348] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:59:26.498848 5434 raft_consensus.cc:729] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.498982 5434 consensus_queue.cc:260] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:59:26.499207 5434 ts_tablet_manager.cc:1398] T d8df290dcc75450b89cf9d3989ed7569 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:26.499377 5434 tablet_bootstrap.cc:492] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.527650 5434 tablet_bootstrap.cc:492] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.527948 5434 tablet_bootstrap.cc:492] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.528738 5434 ts_tablet_manager.cc:1367] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.004s
I20240503 13:59:26.529057 5434 raft_consensus.cc:348] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:59:26.529150 5434 raft_consensus.cc:729] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.529291 5434 consensus_queue.cc:260] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } }
I20240503 13:59:26.529502 5434 ts_tablet_manager.cc:1398] T a35914f78ffe423e91ce866785764b9c P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:26.529644 5434 tablet_bootstrap.cc:492] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.555972 5434 tablet_bootstrap.cc:492] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.556321 5434 tablet_bootstrap.cc:492] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.557416 5434 ts_tablet_manager.cc:1367] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.003s
I20240503 13:59:26.557825 5434 raft_consensus.cc:348] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:59:26.557945 5434 raft_consensus.cc:729] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.558068 5434 consensus_queue.cc:260] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "e59e5d411bc046a28486330e3e4ffb66" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 40971 } }
I20240503 13:59:26.558776 5434 ts_tablet_manager.cc:1398] T 3fd93f79958e489b9028f5123c600f74 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:26.558928 5434 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap starting.
I20240503 13:59:26.576659 2115 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:26.581149 1984 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:26.581861 2377 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:26.585050 693 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:26.591662 5476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:26.594163 5434 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:26.594496 5434 tablet_bootstrap.cc:492] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc: Bootstrap complete.
I20240503 13:59:26.595598 5434 ts_tablet_manager.cc:1367] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent bootstrapping tablet: real 0.037s user 0.025s sys 0.007s
I20240503 13:59:26.595981 5434 raft_consensus.cc:348] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:59:26.596086 5434 raft_consensus.cc:729] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769439c9b5d3418eb0ebd0287954e6cc, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.596213 5434 consensus_queue.cc:260] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769439c9b5d3418eb0ebd0287954e6cc" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33075 } } peers { permanent_uuid: "f58fb372531d42a3bc405510bf5bced4" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 44047 } } peers { permanent_uuid: "37dbd0b8eaae46d59b318fbbb4bab576" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 42209 } }
I20240503 13:59:26.596818 5434 ts_tablet_manager.cc:1398] T eb3d68a0d39042c98d02c9a93307bb88 P 769439c9b5d3418eb0ebd0287954e6cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:26.602931 824 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:26.609911 956 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:26.611374 2246 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:26.651324 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 5410
I20240503 13:59:26.658463 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 632
I20240503 13:59:26.666752 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 763
I20240503 13:59:26.673660 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 894
I20240503 13:59:26.679746 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 1922
I20240503 13:59:26.685815 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 2053
I20240503 13:59:26.691107 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 2184
I20240503 13:59:26.696517 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 2315
I20240503 13:59:26.701740 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 410
2024-05-03T13:59:26Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (49035 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (49035 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240503 13:59:26.739504 398 test_util.cc:268] Using random seed: 548647976
I20240503 13:59:26.740234 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:59:26.740280 398 ts_itest-base.cc:113] --------------
I20240503 13:59:26.740296 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 13:59:26.740317 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 13:59:26.740330 398 ts_itest-base.cc:116] --------------
2024-05-03T13:59:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:59:26Z Disabled control of system clock
I20240503 13:59:26.749243 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37053
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:37053
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 13:59:26.848629 5648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:26.848853 5648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:26.848901 5648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:26.851585 5648 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:26.851641 5648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:59:26.851687 5648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:26.851719 5648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:59:26.851753 5648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:59:26.855006 5648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:37053
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:37053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:26.855823 5648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:26.856827 5648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:26.864125 5653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.864902 5654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:26.865726 5656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:26.866899 5648 server_base.cc:1034] running on GCE node
I20240503 13:59:26.867347 5648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:26.868131 5648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:26.869275 5648 hybrid_clock.cc:648] HybridClock initialized: now 1714744766869252 us; error 38 us; skew 500 ppm
I20240503 13:59:26.871019 5648 webserver.cc:460] Webserver started at http://127.0.99.190:34507/ using document root <none> and password file <none>
I20240503 13:59:26.871582 5648 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:26.871642 5648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:26.871834 5648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:26.873420 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f46ba78ec493489a80a2c8130cb03a90"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-snql"
I20240503 13:59:26.873858 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f46ba78ec493489a80a2c8130cb03a90"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-snql"
I20240503 13:59:26.877032 5648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:26.879024 5662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:26.879909 5648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:26.880014 5648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f46ba78ec493489a80a2c8130cb03a90"
format_stamp: "Formatted at 2024-05-03 13:59:26 on dist-test-slave-snql"
I20240503 13:59:26.880086 5648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:26.900941 5648 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:26.901540 5648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:26.901696 5648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:26.907611 5714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:37053 every 8 connection(s)
I20240503 13:59:26.907596 5648 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:37053
I20240503 13:59:26.908488 5648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:59:26.910816 5715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:26.914664 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 5648
I20240503 13:59:26.914767 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:59:26.915727 5715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90: Bootstrap starting.
I20240503 13:59:26.917690 5715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:26.918325 5715 log.cc:824] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:26.919768 5715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90: No bootstrap required, opened a new log
I20240503 13:59:26.922529 5715 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ba78ec493489a80a2c8130cb03a90" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37053 } }
I20240503 13:59:26.922700 5715 raft_consensus.cc:374] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:26.922763 5715 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46ba78ec493489a80a2c8130cb03a90, State: Initialized, Role: FOLLOWER
I20240503 13:59:26.923194 5715 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ba78ec493489a80a2c8130cb03a90" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37053 } }
I20240503 13:59:26.923350 5715 raft_consensus.cc:388] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:26.923414 5715 raft_consensus.cc:482] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:26.923528 5715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:26.924181 5715 raft_consensus.cc:504] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ba78ec493489a80a2c8130cb03a90" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37053 } }
I20240503 13:59:26.924448 5715 leader_election.cc:302] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f46ba78ec493489a80a2c8130cb03a90; no voters:
I20240503 13:59:26.924679 5715 leader_election.cc:288] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:26.924772 5720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:26.924978 5720 raft_consensus.cc:686] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [term 1 LEADER]: Becoming Leader. State: Replica: f46ba78ec493489a80a2c8130cb03a90, State: Running, Role: LEADER
I20240503 13:59:26.925213 5720 consensus_queue.cc:237] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ba78ec493489a80a2c8130cb03a90" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37053 } }
I20240503 13:59:26.925647 5715 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:26.926862 5722 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f46ba78ec493489a80a2c8130cb03a90. Latest consensus state: current_term: 1 leader_uuid: "f46ba78ec493489a80a2c8130cb03a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ba78ec493489a80a2c8130cb03a90" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37053 } } }
I20240503 13:59:26.926967 5722 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:26.927186 5721 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f46ba78ec493489a80a2c8130cb03a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46ba78ec493489a80a2c8130cb03a90" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 37053 } } }
I20240503 13:59:26.927285 5721 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90 [sys.catalog]: This master's current role is: LEADER
W20240503 13:59:26.929522 5735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 13:59:26.929577 5735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 13:59:26.929641 5732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:26.930280 5732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:26.934463 5732 catalog_manager.cc:1347] Generated new cluster ID: 9d12551ec30a42e484ea1dbdabc764d5
I20240503 13:59:26.934523 5732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:26.948251 5732 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:59:26.949155 5732 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:26.956130 5732 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f46ba78ec493489a80a2c8130cb03a90: Generated new TSK 0
I20240503 13:59:26.956786 5732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:26.968065 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37053
--builtin_ntp_servers=127.0.99.148:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:59:27.072371 5739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:27.072583 5739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:27.072652 5739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:27.075544 5739 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:27.075604 5739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:27.075686 5739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:59:27.078837 5739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37053
--never_fsync=true
--heap_profile_path=/tmp/kudu.5739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:27.079665 5739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:27.080782 5739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:27.089375 5745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.090416 5744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.091367 5747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:27.092226 5739 server_base.cc:1034] running on GCE node
I20240503 13:59:27.092556 5739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:27.093072 5739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:27.094210 5739 hybrid_clock.cc:648] HybridClock initialized: now 1714744767094194 us; error 32 us; skew 500 ppm
I20240503 13:59:27.095883 5739 webserver.cc:460] Webserver started at http://127.0.99.129:41743/ using document root <none> and password file <none>
I20240503 13:59:27.096338 5739 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:27.096385 5739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:27.096537 5739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:27.098018 5739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1b79cdc12ca24f52850bdd5088bb7af9"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.098446 5739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1b79cdc12ca24f52850bdd5088bb7af9"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.101362 5739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 13:59:27.103368 5753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.104269 5739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:27.104379 5739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1b79cdc12ca24f52850bdd5088bb7af9"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.104454 5739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:27.116171 5739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:27.116604 5739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:27.116739 5739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:27.117200 5739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:27.117965 5739 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:27.118012 5739 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.118063 5739 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:27.118095 5739 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.125896 5739 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:35109
I20240503 13:59:27.125943 5866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:35109 every 8 connection(s)
I20240503 13:59:27.126803 5739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:59:27.133771 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 5739
I20240503 13:59:27.133914 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:59:27.135525 5867 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37053
I20240503 13:59:27.135725 5867 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:27.135883 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37053
--builtin_ntp_servers=127.0.99.148:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:59:27.136134 5867 heartbeater.cc:511] Master 127.0.99.190:37053 requested a full tablet report, sending...
I20240503 13:59:27.137378 5678 ts_manager.cc:194] Registered new tserver with Master: 1b79cdc12ca24f52850bdd5088bb7af9 (127.0.99.129:35109)
I20240503 13:59:27.138546 5678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:58407
W20240503 13:59:27.228422 5870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:27.228593 5870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:27.228673 5870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:27.231128 5870 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:27.231182 5870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:27.231280 5870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:59:27.234401 5870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37053
--never_fsync=true
--heap_profile_path=/tmp/kudu.5870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:27.235038 5870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:27.235909 5870 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:27.243726 5878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.244578 5876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.245728 5875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:27.246587 5870 server_base.cc:1034] running on GCE node
I20240503 13:59:27.246894 5870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:27.247327 5870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:27.248454 5870 hybrid_clock.cc:648] HybridClock initialized: now 1714744767248442 us; error 27 us; skew 500 ppm
I20240503 13:59:27.249982 5870 webserver.cc:460] Webserver started at http://127.0.99.130:42009/ using document root <none> and password file <none>
I20240503 13:59:27.250386 5870 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:27.250429 5870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:27.250553 5870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:27.251989 5870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4a99d8856ab74c099b2567c93342c0aa"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.252385 5870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4a99d8856ab74c099b2567c93342c0aa"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.255007 5870 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:27.256729 5884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.257489 5870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:27.257565 5870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4a99d8856ab74c099b2567c93342c0aa"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.257620 5870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:27.271714 5870 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:27.272137 5870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:27.272257 5870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:27.272660 5870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:27.273350 5870 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:27.273389 5870 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.273419 5870 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:27.273437 5870 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.280591 5870 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:45831
I20240503 13:59:27.280615 5997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:45831 every 8 connection(s)
I20240503 13:59:27.281360 5870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:59:27.288851 5998 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37053
I20240503 13:59:27.289038 5998 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:27.289376 5998 heartbeater.cc:511] Master 127.0.99.190:37053 requested a full tablet report, sending...
I20240503 13:59:27.290138 5678 ts_manager.cc:194] Registered new tserver with Master: 4a99d8856ab74c099b2567c93342c0aa (127.0.99.130:45831)
I20240503 13:59:27.290611 5678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:51941
I20240503 13:59:27.291457 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 5870
I20240503 13:59:27.291544 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:59:27.293372 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37053
--builtin_ntp_servers=127.0.99.148:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:59:27.393637 6001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:27.393844 6001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:27.393911 6001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:27.396479 6001 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:27.396536 6001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:27.396616 6001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:59:27.399494 6001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37053
--never_fsync=true
--heap_profile_path=/tmp/kudu.6001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:27.400144 6001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:27.401031 6001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:27.409046 6006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.410228 6009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.410902 6007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:27.411743 6001 server_base.cc:1034] running on GCE node
I20240503 13:59:27.412051 6001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:27.412529 6001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:27.413658 6001 hybrid_clock.cc:648] HybridClock initialized: now 1714744767413649 us; error 39 us; skew 500 ppm
I20240503 13:59:27.415184 6001 webserver.cc:460] Webserver started at http://127.0.99.131:44381/ using document root <none> and password file <none>
I20240503 13:59:27.415661 6001 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:27.415719 6001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:27.415846 6001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:27.417199 6001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "be432784d12a4887b2e0ab8246320c42"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.417589 6001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "be432784d12a4887b2e0ab8246320c42"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.420284 6001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:27.422119 6015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.422995 6001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:27.423079 6001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "be432784d12a4887b2e0ab8246320c42"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.423143 6001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:27.433275 6001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:27.433754 6001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:27.433866 6001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:27.434259 6001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:27.434969 6001 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:27.435011 6001 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.435045 6001 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:27.435065 6001 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.442853 6001 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:38381
I20240503 13:59:27.442898 6128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:38381 every 8 connection(s)
I20240503 13:59:27.443729 6001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:59:27.448570 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 6001
I20240503 13:59:27.448688 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:59:27.450624 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37053
--builtin_ntp_servers=127.0.99.148:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:59:27.451401 6129 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37053
I20240503 13:59:27.451571 6129 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:27.451969 6129 heartbeater.cc:511] Master 127.0.99.190:37053 requested a full tablet report, sending...
I20240503 13:59:27.452914 5678 ts_manager.cc:194] Registered new tserver with Master: be432784d12a4887b2e0ab8246320c42 (127.0.99.131:38381)
I20240503 13:59:27.453394 5678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:39909
W20240503 13:59:27.547551 6132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:27.547757 6132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:27.547824 6132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:27.550726 6132 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:27.550788 6132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:27.550895 6132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:59:27.554348 6132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37053
--never_fsync=true
--heap_profile_path=/tmp/kudu.6132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:27.555173 6132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:27.556430 6132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:27.565173 6138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.566000 6137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.567318 6140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:27.568305 6132 server_base.cc:1034] running on GCE node
I20240503 13:59:27.568663 6132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:27.569187 6132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:27.570330 6132 hybrid_clock.cc:648] HybridClock initialized: now 1714744767570315 us; error 37 us; skew 500 ppm
I20240503 13:59:27.572136 6132 webserver.cc:460] Webserver started at http://127.0.99.132:37637/ using document root <none> and password file <none>
I20240503 13:59:27.572654 6132 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:27.572723 6132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:27.572911 6132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:27.574479 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3b7fdedb93774c019205e52c7dbdd7d3"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.574930 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3b7fdedb93774c019205e52c7dbdd7d3"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.578116 6132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:27.580144 6146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.581054 6132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:27.581163 6132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3b7fdedb93774c019205e52c7dbdd7d3"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.581238 6132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:27.605608 6132 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:27.606256 6132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:27.606379 6132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:27.606856 6132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:27.607774 6132 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:27.607818 6132 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.607856 6132 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:27.607877 6132 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.615733 6132 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:32979
I20240503 13:59:27.615763 6259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:32979 every 8 connection(s)
I20240503 13:59:27.616627 6132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:59:27.624823 6260 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37053
I20240503 13:59:27.625030 6260 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:27.625392 6260 heartbeater.cc:511] Master 127.0.99.190:37053 requested a full tablet report, sending...
I20240503 13:59:27.626204 5678 ts_manager.cc:194] Registered new tserver with Master: 3b7fdedb93774c019205e52c7dbdd7d3 (127.0.99.132:32979)
I20240503 13:59:27.626508 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 6132
I20240503 13:59:27.626605 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:59:27.626801 5678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:38909
I20240503 13:59:27.628508 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37053
--builtin_ntp_servers=127.0.99.148:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:59:27.726080 6263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:27.726287 6263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:27.726374 6263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:27.729152 6263 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:27.729216 6263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:27.729297 6263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:59:27.732271 6263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:37053
--never_fsync=true
--heap_profile_path=/tmp/kudu.6263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:27.733060 6263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:27.734052 6263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:27.741844 6268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.742640 6271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:27.743393 6269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:27.744565 6263 server_base.cc:1034] running on GCE node
I20240503 13:59:27.744861 6263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:27.745296 6263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:27.746428 6263 hybrid_clock.cc:648] HybridClock initialized: now 1714744767746419 us; error 23 us; skew 500 ppm
I20240503 13:59:27.747929 6263 webserver.cc:460] Webserver started at http://127.0.99.133:39229/ using document root <none> and password file <none>
I20240503 13:59:27.748368 6263 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:27.748421 6263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:27.748584 6263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:27.750046 6263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.750442 6263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.753317 6263 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:27.755131 6277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.756007 6263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:27.756085 6263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca"
format_stamp: "Formatted at 2024-05-03 13:59:27 on dist-test-slave-snql"
I20240503 13:59:27.756143 6263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:27.771299 6263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:27.771725 6263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:27.771842 6263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:27.772238 6263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:27.772972 6263 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:27.773016 6263 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.773046 6263 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:27.773064 6263 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:27.780251 6263 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:33875
I20240503 13:59:27.780270 6390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:33875 every 8 connection(s)
I20240503 13:59:27.780969 6263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:59:27.783666 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 6263
I20240503 13:59:27.783766 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:59:27.788815 6391 heartbeater.cc:347] Connected to a master server at 127.0.99.190:37053
I20240503 13:59:27.788996 6391 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:27.789337 6391 heartbeater.cc:511] Master 127.0.99.190:37053 requested a full tablet report, sending...
I20240503 13:59:27.790071 5678 ts_manager.cc:194] Registered new tserver with Master: 17f31b6fbc85430fa05b3bf2bc67d4ca (127.0.99.133:33875)
I20240503 13:59:27.790580 5678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:53937
I20240503 13:59:27.797129 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:59:27.804410 5678 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60962:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 13:59:27.818403 6325 tablet_service.cc:1467] Processing CreateTablet for tablet 6de977bf8b3f46608195838e1db86486 (DEFAULT_TABLE table=TestTable [id=6fe963822b594bae9b20754344abbd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.819149 6325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de977bf8b3f46608195838e1db86486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.819012 5801 tablet_service.cc:1467] Processing CreateTablet for tablet 6de977bf8b3f46608195838e1db86486 (DEFAULT_TABLE table=TestTable [id=6fe963822b594bae9b20754344abbd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.819010 5932 tablet_service.cc:1467] Processing CreateTablet for tablet 6de977bf8b3f46608195838e1db86486 (DEFAULT_TABLE table=TestTable [id=6fe963822b594bae9b20754344abbd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:27.819824 5801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de977bf8b3f46608195838e1db86486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.819824 5932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de977bf8b3f46608195838e1db86486. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:27.824458 6408 tablet_bootstrap.cc:492] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca: Bootstrap starting.
I20240503 13:59:27.824913 6409 tablet_bootstrap.cc:492] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9: Bootstrap starting.
I20240503 13:59:27.825400 6410 tablet_bootstrap.cc:492] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa: Bootstrap starting.
I20240503 13:59:27.826426 6408 tablet_bootstrap.cc:654] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.827289 6408 log.cc:824] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:27.827955 6409 tablet_bootstrap.cc:654] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.828425 6410 tablet_bootstrap.cc:654] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:27.828658 6409 log.cc:824] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:27.828949 6408 tablet_bootstrap.cc:492] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca: No bootstrap required, opened a new log
I20240503 13:59:27.829075 6410 log.cc:824] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:27.829123 6408 ts_tablet_manager.cc:1367] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:59:27.829968 6409 tablet_bootstrap.cc:492] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9: No bootstrap required, opened a new log
I20240503 13:59:27.830111 6409 ts_tablet_manager.cc:1367] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:27.830448 6410 tablet_bootstrap.cc:492] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa: No bootstrap required, opened a new log
I20240503 13:59:27.830580 6410 ts_tablet_manager.cc:1367] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 13:59:27.832439 6408 raft_consensus.cc:348] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33875 } } peers { permanent_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35109 } } peers { permanent_uuid: "4a99d8856ab74c099b2567c93342c0aa" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45831 } }
I20240503 13:59:27.832638 6408 raft_consensus.cc:374] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.832564 6409 raft_consensus.cc:348] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33875 } } peers { permanent_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35109 } } peers { permanent_uuid: "4a99d8856ab74c099b2567c93342c0aa" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45831 } }
I20240503 13:59:27.832700 6408 raft_consensus.cc:729] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f31b6fbc85430fa05b3bf2bc67d4ca, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.832756 6409 raft_consensus.cc:374] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.832818 6409 raft_consensus.cc:729] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b79cdc12ca24f52850bdd5088bb7af9, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.833205 6409 consensus_queue.cc:260] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33875 } } peers { permanent_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35109 } } peers { permanent_uuid: "4a99d8856ab74c099b2567c93342c0aa" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45831 } }
I20240503 13:59:27.833158 6408 consensus_queue.cc:260] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33875 } } peers { permanent_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35109 } } peers { permanent_uuid: "4a99d8856ab74c099b2567c93342c0aa" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45831 } }
I20240503 13:59:27.833912 6408 ts_tablet_manager.cc:1398] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 13:59:27.833927 6409 ts_tablet_manager.cc:1398] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:27.833855 6410 raft_consensus.cc:348] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33875 } } peers { permanent_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35109 } } peers { permanent_uuid: "4a99d8856ab74c099b2567c93342c0aa" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45831 } }
I20240503 13:59:27.834017 6410 raft_consensus.cc:374] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:27.834071 6410 raft_consensus.cc:729] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a99d8856ab74c099b2567c93342c0aa, State: Initialized, Role: FOLLOWER
I20240503 13:59:27.834208 5867 heartbeater.cc:503] Master 127.0.99.190:37053 was elected leader, sending a full tablet report...
I20240503 13:59:27.834357 6410 consensus_queue.cc:260] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33875 } } peers { permanent_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35109 } } peers { permanent_uuid: "4a99d8856ab74c099b2567c93342c0aa" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45831 } }
I20240503 13:59:27.834842 5998 heartbeater.cc:503] Master 127.0.99.190:37053 was elected leader, sending a full tablet report...
I20240503 13:59:27.834843 6410 ts_tablet_manager.cc:1398] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 13:59:27.834995 6391 heartbeater.cc:503] Master 127.0.99.190:37053 was elected leader, sending a full tablet report...
W20240503 13:59:27.877976 5868 tablet.cc:2369] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:27.965090 6416 raft_consensus.cc:482] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:27.965292 6416 raft_consensus.cc:504] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33875 } } peers { permanent_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35109 } } peers { permanent_uuid: "4a99d8856ab74c099b2567c93342c0aa" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45831 } }
I20240503 13:59:27.966295 6416 leader_election.cc:288] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17f31b6fbc85430fa05b3bf2bc67d4ca (127.0.99.133:33875), 1b79cdc12ca24f52850bdd5088bb7af9 (127.0.99.129:35109)
I20240503 13:59:27.970335 6345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de977bf8b3f46608195838e1db86486" candidate_uuid: "4a99d8856ab74c099b2567c93342c0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" is_pre_election: true
I20240503 13:59:27.970443 5821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de977bf8b3f46608195838e1db86486" candidate_uuid: "4a99d8856ab74c099b2567c93342c0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" is_pre_election: true
I20240503 13:59:27.970598 6345 raft_consensus.cc:2454] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a99d8856ab74c099b2567c93342c0aa in term 0.
I20240503 13:59:27.970707 5821 raft_consensus.cc:2454] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a99d8856ab74c099b2567c93342c0aa in term 0.
I20240503 13:59:27.971010 5888 leader_election.cc:302] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f31b6fbc85430fa05b3bf2bc67d4ca, 4a99d8856ab74c099b2567c93342c0aa; no voters:
I20240503 13:59:27.971225 6416 raft_consensus.cc:2781] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:27.971302 6416 raft_consensus.cc:482] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:27.971346 6416 raft_consensus.cc:3037] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.972242 6416 raft_consensus.cc:504] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33875 } } peers { permanent_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35109 } } peers { permanent_uuid: "4a99d8856ab74c099b2567c93342c0aa" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45831 } }
I20240503 13:59:27.972549 6416 leader_election.cc:288] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [CANDIDATE]: Term 1 election: Requested vote from peers 17f31b6fbc85430fa05b3bf2bc67d4ca (127.0.99.133:33875), 1b79cdc12ca24f52850bdd5088bb7af9 (127.0.99.129:35109)
I20240503 13:59:27.972887 6345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de977bf8b3f46608195838e1db86486" candidate_uuid: "4a99d8856ab74c099b2567c93342c0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca"
I20240503 13:59:27.972993 6345 raft_consensus.cc:3037] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.973027 5821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de977bf8b3f46608195838e1db86486" candidate_uuid: "4a99d8856ab74c099b2567c93342c0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b79cdc12ca24f52850bdd5088bb7af9"
I20240503 13:59:27.973150 5821 raft_consensus.cc:3037] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:27.973804 6345 raft_consensus.cc:2454] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a99d8856ab74c099b2567c93342c0aa in term 1.
I20240503 13:59:27.973925 5821 raft_consensus.cc:2454] T 6de977bf8b3f46608195838e1db86486 P 1b79cdc12ca24f52850bdd5088bb7af9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a99d8856ab74c099b2567c93342c0aa in term 1.
I20240503 13:59:27.974077 5888 leader_election.cc:302] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f31b6fbc85430fa05b3bf2bc67d4ca, 4a99d8856ab74c099b2567c93342c0aa; no voters:
I20240503 13:59:27.974246 6416 raft_consensus.cc:2781] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:27.974426 6416 raft_consensus.cc:686] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [term 1 LEADER]: Becoming Leader. State: Replica: 4a99d8856ab74c099b2567c93342c0aa, State: Running, Role: LEADER
I20240503 13:59:27.974704 6416 consensus_queue.cc:237] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33875 } } peers { permanent_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35109 } } peers { permanent_uuid: "4a99d8856ab74c099b2567c93342c0aa" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45831 } }
I20240503 13:59:27.977331 5679 catalog_manager.cc:5484] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a99d8856ab74c099b2567c93342c0aa (127.0.99.130). New cstate: current_term: 1 leader_uuid: "4a99d8856ab74c099b2567c93342c0aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33875 } } peers { permanent_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35109 } } peers { permanent_uuid: "4a99d8856ab74c099b2567c93342c0aa" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 45831 } } }
I20240503 13:59:27.999940 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 13:59:28.001475 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1b79cdc12ca24f52850bdd5088bb7af9 to finish bootstrapping
I20240503 13:59:28.006645 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4a99d8856ab74c099b2567c93342c0aa to finish bootstrapping
I20240503 13:59:28.010816 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver be432784d12a4887b2e0ab8246320c42 to finish bootstrapping
I20240503 13:59:28.015897 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3b7fdedb93774c019205e52c7dbdd7d3 to finish bootstrapping
I20240503 13:59:28.020684 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 17f31b6fbc85430fa05b3bf2bc67d4ca to finish bootstrapping
W20240503 13:59:28.031721 6392 tablet.cc:2369] T 6de977bf8b3f46608195838e1db86486 P 17f31b6fbc85430fa05b3bf2bc67d4ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:59:28.032529 5999 tablet.cc:2369] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:28.196414 6194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.198184 5801 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.198231 6325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.200820 6063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.212679 5932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:28.284907 5932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:28.285173 6194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.285604 6325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.287770 5801 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.304337 6063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.454881 6129 heartbeater.cc:503] Master 127.0.99.190:37053 was elected leader, sending a full tablet report...
I20240503 13:59:28.485687 6194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.487272 6325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.488207 5801 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.489189 5932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:28.497130 6063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.568732 6325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.569865 6063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.570693 6194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.570724 5932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:28.581930 5801 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.593380 6557 consensus_queue.cc:1035] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b79cdc12ca24f52850bdd5088bb7af9" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 35109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.597046 6557 consensus_queue.cc:1035] T 6de977bf8b3f46608195838e1db86486 P 4a99d8856ab74c099b2567c93342c0aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "17f31b6fbc85430fa05b3bf2bc67d4ca" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 33875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:28.628764 6260 heartbeater.cc:503] Master 127.0.99.190:37053 was elected leader, sending a full tablet report...
I20240503 13:59:28.665323 5801 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.666306 6194 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.667094 6325 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.667409 6063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:28.668188 5932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:28.703661 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 5739
I20240503 13:59:28.709609 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 5870
I20240503 13:59:28.715262 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 6001
I20240503 13:59:28.720208 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 6132
I20240503 13:59:28.725351 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 6263
I20240503 13:59:28.730742 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 5648
2024-05-03T13:59:28Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (2014 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2014 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240503 13:59:28.754242 398 test_util.cc:268] Using random seed: 550662719
I20240503 13:59:28.754911 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 13:59:28.754949 398 ts_itest-base.cc:113] --------------
I20240503 13:59:28.754959 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 13:59:28.754969 398 ts_itest-base.cc:115] 2 replicas per TS
I20240503 13:59:28.754977 398 ts_itest-base.cc:116] --------------
2024-05-03T13:59:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T13:59:28Z Disabled control of system clock
I20240503 13:59:28.762406 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42439
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:42439
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 13:59:28.855705 6605 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 13:59:28.855945 6605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:28.855996 6605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:28.856058 6605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:28.858712 6605 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:28.858788 6605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 13:59:28.858827 6605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:28.858863 6605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 13:59:28.858901 6605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 13:59:28.862280 6605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:42439
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:28.863111 6605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:28.864154 6605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:28.871351 6610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:28.872118 6611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:28.872956 6613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:28.874110 6605 server_base.cc:1034] running on GCE node
I20240503 13:59:28.874521 6605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:28.875317 6605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:28.876456 6605 hybrid_clock.cc:648] HybridClock initialized: now 1714744768876448 us; error 23 us; skew 500 ppm
I20240503 13:59:28.878161 6605 webserver.cc:460] Webserver started at http://127.0.99.190:39937/ using document root <none> and password file <none>
I20240503 13:59:28.878634 6605 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:28.878690 6605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:28.878840 6605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:28.880321 6605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "11a95c76ed294a80a8da622eab79df3a"
format_stamp: "Formatted at 2024-05-03 13:59:28 on dist-test-slave-snql"
I20240503 13:59:28.880724 6605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "11a95c76ed294a80a8da622eab79df3a"
format_stamp: "Formatted at 2024-05-03 13:59:28 on dist-test-slave-snql"
I20240503 13:59:28.883728 6605 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 13:59:28.885615 6619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:28.886471 6605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:28.886571 6605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "11a95c76ed294a80a8da622eab79df3a"
format_stamp: "Formatted at 2024-05-03 13:59:28 on dist-test-slave-snql"
I20240503 13:59:28.886641 6605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:28.900213 6605 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:28.900708 6605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:28.900838 6605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:28.906798 6605 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:42439
I20240503 13:59:28.906821 6671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:42439 every 8 connection(s)
I20240503 13:59:28.907642 6605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 13:59:28.909885 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:28.914402 6672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a: Bootstrap starting.
I20240503 13:59:28.916409 6672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:28.916998 6672 log.cc:824] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:28.917249 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 6605
I20240503 13:59:28.917343 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 13:59:28.918607 6672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a: No bootstrap required, opened a new log
I20240503 13:59:28.921697 6672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a95c76ed294a80a8da622eab79df3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42439 } }
I20240503 13:59:28.921877 6672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:28.921933 6672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a95c76ed294a80a8da622eab79df3a, State: Initialized, Role: FOLLOWER
I20240503 13:59:28.922339 6672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a95c76ed294a80a8da622eab79df3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42439 } }
I20240503 13:59:28.922446 6672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:28.922503 6672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:28.922616 6672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:28.923480 6672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a95c76ed294a80a8da622eab79df3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42439 } }
I20240503 13:59:28.923847 6672 leader_election.cc:302] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11a95c76ed294a80a8da622eab79df3a; no voters:
I20240503 13:59:28.924119 6672 leader_election.cc:288] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:28.924465 6677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:28.924682 6677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [term 1 LEADER]: Becoming Leader. State: Replica: 11a95c76ed294a80a8da622eab79df3a, State: Running, Role: LEADER
I20240503 13:59:28.924983 6677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a95c76ed294a80a8da622eab79df3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42439 } }
I20240503 13:59:28.925164 6672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:28.926303 6678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11a95c76ed294a80a8da622eab79df3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a95c76ed294a80a8da622eab79df3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42439 } } }
I20240503 13:59:28.926395 6678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:28.926641 6679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11a95c76ed294a80a8da622eab79df3a. Latest consensus state: current_term: 1 leader_uuid: "11a95c76ed294a80a8da622eab79df3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a95c76ed294a80a8da622eab79df3a" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42439 } } }
I20240503 13:59:28.926724 6679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:28.926980 6686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:28.929618 6686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:28.933951 6686 catalog_manager.cc:1347] Generated new cluster ID: 4829107a897c4d49b9b2a935337aab82
I20240503 13:59:28.933996 6686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:28.952399 6686 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:59:28.953198 6686 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:28.960059 6686 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 11a95c76ed294a80a8da622eab79df3a: Generated new TSK 0
I20240503 13:59:28.960636 6686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:28.968503 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:59:29.070396 6696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:29.070598 6696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:29.070667 6696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:29.073634 6696 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:29.073696 6696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:29.073783 6696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 13:59:29.077091 6696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.6696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:29.077803 6696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:29.078809 6696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:29.087412 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.088083 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.089257 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:29.090291 6696 server_base.cc:1034] running on GCE node
I20240503 13:59:29.090644 6696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:29.091198 6696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:29.092368 6696 hybrid_clock.cc:648] HybridClock initialized: now 1714744769092349 us; error 36 us; skew 500 ppm
I20240503 13:59:29.094156 6696 webserver.cc:460] Webserver started at http://127.0.99.129:33033/ using document root <none> and password file <none>
I20240503 13:59:29.094660 6696 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:29.094712 6696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:29.094880 6696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:29.096401 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "791cddb67b7e415aad7edb5151037418"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.096818 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "791cddb67b7e415aad7edb5151037418"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.099915 6696 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 13:59:29.101756 6710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.102682 6696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:29.102785 6696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "791cddb67b7e415aad7edb5151037418"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.102866 6696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:29.116323 6696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:29.116848 6696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:29.116986 6696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:29.117456 6696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:29.118234 6696 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:29.118279 6696 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.118330 6696 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:29.118359 6696 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.126147 6696 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:43893
I20240503 13:59:29.126188 6823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:43893 every 8 connection(s)
I20240503 13:59:29.126986 6696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 13:59:29.134307 6824 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:29.134330 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 6696
I20240503 13:59:29.134433 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 13:59:29.134495 6824 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:29.134866 6824 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:29.136268 6632 ts_manager.cc:194] Registered new tserver with Master: 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:29.136396 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:59:29.137367 6632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:49857
W20240503 13:59:29.234503 6827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:29.234694 6827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:29.234733 6827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:29.237579 6827 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:29.237635 6827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:29.237697 6827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 13:59:29.241055 6827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.6827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:29.241864 6827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:29.242971 6827 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:29.251492 6835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.252540 6832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.253414 6833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:29.254249 6827 server_base.cc:1034] running on GCE node
I20240503 13:59:29.254581 6827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:29.255172 6827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:29.256304 6827 hybrid_clock.cc:648] HybridClock initialized: now 1714744769256287 us; error 33 us; skew 500 ppm
I20240503 13:59:29.258034 6827 webserver.cc:460] Webserver started at http://127.0.99.130:43399/ using document root <none> and password file <none>
I20240503 13:59:29.258626 6827 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:29.258682 6827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:29.258864 6827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:29.260476 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "52a58bb8fe574f5ab1485c59bf994570"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.260917 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "52a58bb8fe574f5ab1485c59bf994570"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.264268 6827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 13:59:29.266386 6841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.267364 6827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 13:59:29.267468 6827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "52a58bb8fe574f5ab1485c59bf994570"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.267546 6827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:29.281047 6827 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:29.281608 6827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:29.281752 6827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:29.282259 6827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:29.283109 6827 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:29.283162 6827 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.283221 6827 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:29.283288 6827 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.291314 6827 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:33439
I20240503 13:59:29.291332 6954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:33439 every 8 connection(s)
I20240503 13:59:29.292203 6827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 13:59:29.293543 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 6827
I20240503 13:59:29.293658 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 13:59:29.295764 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:59:29.300920 6955 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:29.301131 6955 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:29.301542 6955 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:29.302429 6632 ts_manager.cc:194] Registered new tserver with Master: 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:29.302970 6632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:40009
W20240503 13:59:29.397399 6958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:29.397601 6958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:29.397668 6958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:29.400152 6958 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:29.400207 6958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:29.400282 6958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 13:59:29.403545 6958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.6958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:29.404223 6958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:29.405129 6958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:29.412907 6966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.413811 6963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.414458 6964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:29.415606 6958 server_base.cc:1034] running on GCE node
I20240503 13:59:29.415910 6958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:29.416357 6958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:29.417496 6958 hybrid_clock.cc:648] HybridClock initialized: now 1714744769417483 us; error 28 us; skew 500 ppm
I20240503 13:59:29.419100 6958 webserver.cc:460] Webserver started at http://127.0.99.131:40165/ using document root <none> and password file <none>
I20240503 13:59:29.419586 6958 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:29.419636 6958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:29.419785 6958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:29.421168 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "14c195eab0b34945afd10f3291ed080b"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.421577 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "14c195eab0b34945afd10f3291ed080b"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.424587 6958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:29.426614 6972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.427527 6958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:29.427610 6958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "14c195eab0b34945afd10f3291ed080b"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.427686 6958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:29.451306 6958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:29.451867 6958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:29.452006 6958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:29.452543 6958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:29.453388 6958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:29.453429 6958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.453469 6958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:29.453496 6958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.461468 6958 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:40699
I20240503 13:59:29.461511 7085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:40699 every 8 connection(s)
I20240503 13:59:29.462332 6958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 13:59:29.462584 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 6958
I20240503 13:59:29.462714 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 13:59:29.464807 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:59:29.470232 7086 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:29.470415 7086 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:29.470804 7086 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:29.471655 6632 ts_manager.cc:194] Registered new tserver with Master: 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:29.472152 6632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:55145
W20240503 13:59:29.561332 7089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:29.561506 7089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:29.561570 7089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:29.564431 7089 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:29.564488 7089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:29.564575 7089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:59:29.567713 7089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.7089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:29.568401 7089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:29.569391 7089 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:29.577505 7095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.578191 7094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.579306 7097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:29.580163 7089 server_base.cc:1034] running on GCE node
I20240503 13:59:29.580432 7089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:29.580871 7089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:29.582005 7089 hybrid_clock.cc:648] HybridClock initialized: now 1714744769581985 us; error 33 us; skew 500 ppm
I20240503 13:59:29.591303 7089 webserver.cc:460] Webserver started at http://127.0.99.132:44053/ using document root <none> and password file <none>
I20240503 13:59:29.591739 7089 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:29.591789 7089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:29.591922 7089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:29.593395 7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "40e8eacd02b44bb8bbec0a1dc205a671"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.593755 7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "40e8eacd02b44bb8bbec0a1dc205a671"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.596568 7089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 13:59:29.598284 7103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.599077 7089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:29.599159 7089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "40e8eacd02b44bb8bbec0a1dc205a671"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.599216 7089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:29.612136 7089 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:29.612622 7089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:29.612737 7089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:29.613162 7089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:29.613914 7089 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:29.613967 7089 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.614006 7089 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:29.614025 7089 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.621642 7089 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:41965
I20240503 13:59:29.621662 7216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:41965 every 8 connection(s)
I20240503 13:59:29.622496 7089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:59:29.629876 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 7089
I20240503 13:59:29.630013 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 13:59:29.630749 7217 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:29.630971 7217 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:29.631501 7217 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:29.632151 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:59:29.632408 6632 ts_manager.cc:194] Registered new tserver with Master: 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132:41965)
I20240503 13:59:29.632917 6632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:60849
W20240503 13:59:29.728999 7220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:29.729183 7220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:29.729223 7220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:29.732080 7220 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:29.732159 7220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:29.732221 7220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:59:29.735423 7220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.7220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:29.736183 7220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:29.737138 7220 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:29.744987 7226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.745738 7225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.746831 7228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:29.747786 7220 server_base.cc:1034] running on GCE node
I20240503 13:59:29.748111 7220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:29.748569 7220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:29.749704 7220 hybrid_clock.cc:648] HybridClock initialized: now 1714744769749699 us; error 29 us; skew 500 ppm
I20240503 13:59:29.751431 7220 webserver.cc:460] Webserver started at http://127.0.99.133:40171/ using document root <none> and password file <none>
I20240503 13:59:29.751940 7220 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:29.752004 7220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:29.752190 7220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:29.753808 7220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.754269 7220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.757187 7220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 13:59:29.759001 7234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.759894 7220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240503 13:59:29.759994 7220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.760072 7220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:29.788502 7220 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:29.789057 7220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:29.789206 7220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:29.789670 7220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:29.790441 7220 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:29.790486 7220 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.790539 7220 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:29.790563 7220 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.798349 7220 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39967
I20240503 13:59:29.798374 7347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39967 every 8 connection(s)
I20240503 13:59:29.799098 7220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:59:29.806509 7348 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:29.806695 7348 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:29.807083 7348 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:29.807448 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 7220
I20240503 13:59:29.807560 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 13:59:29.808039 6632 ts_manager.cc:194] Registered new tserver with Master: 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133:39967)
I20240503 13:59:29.808702 6632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:39035
I20240503 13:59:29.809584 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:59:29.909905 7351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:29.910101 7351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:29.910142 7351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:29.912997 7351 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:29.913048 7351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:29.913128 7351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:59:29.916455 7351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.7351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:29.917146 7351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:29.918085 7351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:29.926785 7359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.927795 7357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:29.928592 7356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:29.929436 7351 server_base.cc:1034] running on GCE node
I20240503 13:59:29.929744 7351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:29.930261 7351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:29.931402 7351 hybrid_clock.cc:648] HybridClock initialized: now 1714744769931389 us; error 25 us; skew 500 ppm
I20240503 13:59:29.933180 7351 webserver.cc:460] Webserver started at http://127.0.99.134:43919/ using document root <none> and password file <none>
I20240503 13:59:29.933696 7351 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:29.933750 7351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:29.933936 7351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:29.935508 7351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "083b1c5c79a04af899eea33ba9f4502f"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.935961 7351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "083b1c5c79a04af899eea33ba9f4502f"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.938926 7351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240503 13:59:29.940951 7365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.941850 7351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:29.941948 7351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "083b1c5c79a04af899eea33ba9f4502f"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:29.942023 7351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:29.961796 7351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:29.962381 7351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:29.962529 7351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:29.963053 7351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:29.963946 7351 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:29.963995 7351 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.964051 7351 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:29.964083 7351 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:29.972142 7351 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:44579
I20240503 13:59:29.972204 7478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:44579 every 8 connection(s)
I20240503 13:59:29.973026 7351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:59:29.974525 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 7351
I20240503 13:59:29.974642 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 13:59:29.976747 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:59:29.981341 7479 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:29.981520 7479 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:29.981906 7479 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:29.982813 6632 ts_manager.cc:194] Registered new tserver with Master: 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579)
I20240503 13:59:29.983369 6632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:58987
W20240503 13:59:30.080065 7482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:30.080296 7482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:30.080367 7482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:30.083338 7482 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:30.083405 7482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:30.083492 7482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:59:30.086692 7482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.7482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:30.087580 7482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:30.088690 7482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:30.097306 7488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:30.098433 7487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:30.099318 7490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:30.100124 7482 server_base.cc:1034] running on GCE node
I20240503 13:59:30.100474 7482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:30.100958 7482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:30.102106 7482 hybrid_clock.cc:648] HybridClock initialized: now 1714744770102093 us; error 31 us; skew 500 ppm
I20240503 13:59:30.103986 7482 webserver.cc:460] Webserver started at http://127.0.99.135:43613/ using document root <none> and password file <none>
I20240503 13:59:30.104483 7482 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:30.104539 7482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:30.104722 7482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:30.106352 7482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "4f82135907754dd8b69d08d6b7eae847"
format_stamp: "Formatted at 2024-05-03 13:59:30 on dist-test-slave-snql"
I20240503 13:59:30.106842 7482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "4f82135907754dd8b69d08d6b7eae847"
format_stamp: "Formatted at 2024-05-03 13:59:30 on dist-test-slave-snql"
I20240503 13:59:30.110071 7482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 13:59:30.112102 7496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:30.113023 7482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:30.113121 7482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4f82135907754dd8b69d08d6b7eae847"
format_stamp: "Formatted at 2024-05-03 13:59:30 on dist-test-slave-snql"
I20240503 13:59:30.113198 7482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:30.137915 7482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:30.138466 7482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:30.138602 7482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:30.139070 7482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:30.139688 6824 heartbeater.cc:503] Master 127.0.99.190:42439 was elected leader, sending a full tablet report...
I20240503 13:59:30.140012 7482 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:30.140058 7482 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:30.140103 7482 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:30.140132 7482 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:30.148911 7482 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:34409
I20240503 13:59:30.148938 7609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:34409 every 8 connection(s)
I20240503 13:59:30.149703 7482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:59:30.152532 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 7482
I20240503 13:59:30.152657 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 13:59:30.157841 7610 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:30.158038 7610 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:30.158427 7610 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:30.159207 6632 ts_manager.cc:194] Registered new tserver with Master: 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135:34409)
I20240503 13:59:30.159703 6632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:33937
I20240503 13:59:30.166291 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 13:59:30.174221 6632 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60082:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 13:59:30.189774 7151 tablet_service.cc:1467] Processing CreateTablet for tablet 80d845691ee64bf3a5d6c2df59842a54 (DEFAULT_TABLE table=TestTable [id=7284d6ec66314b6aa59c53dfab265168]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:30.189771 7413 tablet_service.cc:1467] Processing CreateTablet for tablet 80d845691ee64bf3a5d6c2df59842a54 (DEFAULT_TABLE table=TestTable [id=7284d6ec66314b6aa59c53dfab265168]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:30.190608 7151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d845691ee64bf3a5d6c2df59842a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.190608 7413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d845691ee64bf3a5d6c2df59842a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:30.195282 7626 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap starting.
I20240503 13:59:30.195649 7627 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap starting.
I20240503 13:59:30.197362 7626 tablet_bootstrap.cc:654] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.197364 7627 tablet_bootstrap.cc:654] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:30.198007 7626 log.cc:824] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:30.198041 7627 log.cc:824] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:30.199450 7626 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: No bootstrap required, opened a new log
I20240503 13:59:30.199460 7627 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: No bootstrap required, opened a new log
I20240503 13:59:30.199601 7626 ts_tablet_manager.cc:1367] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:30.199609 7627 ts_tablet_manager.cc:1367] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:30.201853 7626 raft_consensus.cc:348] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:30.202016 7626 raft_consensus.cc:374] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.202056 7626 raft_consensus.cc:729] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e8eacd02b44bb8bbec0a1dc205a671, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.202368 7627 raft_consensus.cc:348] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:30.202440 7626 consensus_queue.cc:260] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:30.202515 7627 raft_consensus.cc:374] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:30.202582 7627 raft_consensus.cc:729] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083b1c5c79a04af899eea33ba9f4502f, State: Initialized, Role: FOLLOWER
I20240503 13:59:30.202942 7627 consensus_queue.cc:260] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:30.203159 7626 ts_tablet_manager.cc:1398] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:30.203533 7217 heartbeater.cc:503] Master 127.0.99.190:42439 was elected leader, sending a full tablet report...
I20240503 13:59:30.203702 7627 ts_tablet_manager.cc:1398] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:30.204005 7479 heartbeater.cc:503] Master 127.0.99.190:42439 was elected leader, sending a full tablet report...
W20240503 13:59:30.223809 7480 tablet.cc:2369] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:30.245162 7631 raft_consensus.cc:482] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:30.245311 7631 raft_consensus.cc:504] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:30.246146 7631 leader_election.cc:288] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132:41965)
I20240503 13:59:30.249363 7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d845691ee64bf3a5d6c2df59842a54" candidate_uuid: "083b1c5c79a04af899eea33ba9f4502f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" is_pre_election: true
I20240503 13:59:30.249610 7171 raft_consensus.cc:2454] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 083b1c5c79a04af899eea33ba9f4502f in term 0.
I20240503 13:59:30.250008 7367 leader_election.cc:302] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 083b1c5c79a04af899eea33ba9f4502f, 40e8eacd02b44bb8bbec0a1dc205a671; no voters:
I20240503 13:59:30.250226 7631 raft_consensus.cc:2781] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:30.250299 7631 raft_consensus.cc:482] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:30.250353 7631 raft_consensus.cc:3037] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.251175 7631 raft_consensus.cc:504] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:30.251513 7631 leader_election.cc:288] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [CANDIDATE]: Term 1 election: Requested vote from peers 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132:41965)
I20240503 13:59:30.251824 7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d845691ee64bf3a5d6c2df59842a54" candidate_uuid: "083b1c5c79a04af899eea33ba9f4502f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e8eacd02b44bb8bbec0a1dc205a671"
I20240503 13:59:30.251947 7171 raft_consensus.cc:3037] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:30.252776 7171 raft_consensus.cc:2454] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 083b1c5c79a04af899eea33ba9f4502f in term 1.
I20240503 13:59:30.253015 7367 leader_election.cc:302] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 083b1c5c79a04af899eea33ba9f4502f, 40e8eacd02b44bb8bbec0a1dc205a671; no voters:
I20240503 13:59:30.253172 7631 raft_consensus.cc:2781] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:30.253391 7631 raft_consensus.cc:686] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 1 LEADER]: Becoming Leader. State: Replica: 083b1c5c79a04af899eea33ba9f4502f, State: Running, Role: LEADER
I20240503 13:59:30.253671 7631 consensus_queue.cc:237] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:30.256130 6636 catalog_manager.cc:5484] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f reported cstate change: term changed from 0 to 1, leader changed from <none> to 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134). New cstate: current_term: 1 leader_uuid: "083b1c5c79a04af899eea33ba9f4502f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:30.261375 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 13:59:30.262501 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 791cddb67b7e415aad7edb5151037418 to finish bootstrapping
I20240503 13:59:30.267582 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 52a58bb8fe574f5ab1485c59bf994570 to finish bootstrapping
I20240503 13:59:30.272657 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 14c195eab0b34945afd10f3291ed080b to finish bootstrapping
I20240503 13:59:30.277014 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 40e8eacd02b44bb8bbec0a1dc205a671 to finish bootstrapping
I20240503 13:59:30.281092 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4c5b7c28dd2a4920a2b4e97d1afe2384 to finish bootstrapping
I20240503 13:59:30.285873 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 083b1c5c79a04af899eea33ba9f4502f to finish bootstrapping
I20240503 13:59:30.289744 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4f82135907754dd8b69d08d6b7eae847 to finish bootstrapping
I20240503 13:59:30.294569 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 7089
I20240503 13:59:30.300591 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 7220
I20240503 13:59:30.304690 6955 heartbeater.cc:503] Master 127.0.99.190:42439 was elected leader, sending a full tablet report...
I20240503 13:59:30.306732 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 7351
I20240503 13:59:30.312054 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 7482
I20240503 13:59:30.473762 7086 heartbeater.cc:503] Master 127.0.99.190:42439 was elected leader, sending a full tablet report...
I20240503 13:59:34.068154 6636 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60082:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:34.080605 6889 tablet_service.cc:1467] Processing CreateTablet for tablet 527c7e7cf4f24ce3bed276c1860aa041 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9d13cbf321a4a76b3f59808a6da0969]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.080610 7020 tablet_service.cc:1467] Processing CreateTablet for tablet 0a871746624e442597588274959d5e24 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9d13cbf321a4a76b3f59808a6da0969]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.081069 6758 tablet_service.cc:1467] Processing CreateTablet for tablet 0a871746624e442597588274959d5e24 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9d13cbf321a4a76b3f59808a6da0969]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.081524 7020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a871746624e442597588274959d5e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.081506 6888 tablet_service.cc:1467] Processing CreateTablet for tablet 6409f411c5d24e97b34aa9cce7c5a5ea (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9d13cbf321a4a76b3f59808a6da0969]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.081334 6757 tablet_service.cc:1467] Processing CreateTablet for tablet 527c7e7cf4f24ce3bed276c1860aa041 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9d13cbf321a4a76b3f59808a6da0969]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.081935 6758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a871746624e442597588274959d5e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.080605 7019 tablet_service.cc:1467] Processing CreateTablet for tablet 6409f411c5d24e97b34aa9cce7c5a5ea (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9d13cbf321a4a76b3f59808a6da0969]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.082203 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6409f411c5d24e97b34aa9cce7c5a5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.082969 6757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527c7e7cf4f24ce3bed276c1860aa041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.081522 6889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527c7e7cf4f24ce3bed276c1860aa041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.084530 6888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6409f411c5d24e97b34aa9cce7c5a5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.087082 7650 tablet_bootstrap.cc:492] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b: Bootstrap starting.
I20240503 13:59:34.087584 7651 tablet_bootstrap.cc:492] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: Bootstrap starting.
I20240503 13:59:34.089048 7650 tablet_bootstrap.cc:654] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.089388 7651 tablet_bootstrap.cc:654] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.089828 7650 log.cc:824] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:34.090083 7651 log.cc:824] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:34.090636 7652 tablet_bootstrap.cc:492] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570: Bootstrap starting.
I20240503 13:59:34.091761 7651 tablet_bootstrap.cc:492] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: No bootstrap required, opened a new log
I20240503 13:59:34.091761 7650 tablet_bootstrap.cc:492] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b: No bootstrap required, opened a new log
I20240503 13:59:34.091922 7651 ts_tablet_manager.cc:1367] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 13:59:34.091922 7650 ts_tablet_manager.cc:1367] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:59:34.092223 7652 tablet_bootstrap.cc:654] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.093217 7652 log.cc:824] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:34.094424 7652 tablet_bootstrap.cc:492] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570: No bootstrap required, opened a new log
I20240503 13:59:34.094563 7652 ts_tablet_manager.cc:1367] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 13:59:34.095444 7651 raft_consensus.cc:348] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.095444 7650 raft_consensus.cc:348] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.095621 7650 raft_consensus.cc:374] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.095621 7651 raft_consensus.cc:374] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.095682 7651 raft_consensus.cc:729] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.095696 7650 raft_consensus.cc:729] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.096099 7650 consensus_queue.cc:260] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.096099 7651 consensus_queue.cc:260] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.096817 7650 ts_tablet_manager.cc:1398] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 13:59:34.097012 7650 tablet_bootstrap.cc:492] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b: Bootstrap starting.
I20240503 13:59:34.097754 7650 tablet_bootstrap.cc:654] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.097973 7652 raft_consensus.cc:348] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.098186 7652 raft_consensus.cc:374] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.098301 7652 raft_consensus.cc:729] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.098628 7652 consensus_queue.cc:260] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.099489 7652 ts_tablet_manager.cc:1398] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 13:59:34.099684 7652 tablet_bootstrap.cc:492] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570: Bootstrap starting.
I20240503 13:59:34.100416 7651 ts_tablet_manager.cc:1398] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 13:59:34.100638 7651 tablet_bootstrap.cc:492] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418: Bootstrap starting.
I20240503 13:59:34.100891 7652 tablet_bootstrap.cc:654] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.101740 7651 tablet_bootstrap.cc:654] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.102187 7652 tablet_bootstrap.cc:492] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570: No bootstrap required, opened a new log
I20240503 13:59:34.102260 7652 ts_tablet_manager.cc:1367] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:34.102612 7651 tablet_bootstrap.cc:492] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418: No bootstrap required, opened a new log
I20240503 13:59:34.102670 7651 ts_tablet_manager.cc:1367] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:34.102669 7652 raft_consensus.cc:348] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.102762 7652 raft_consensus.cc:374] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.102803 7652 raft_consensus.cc:729] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.102902 7652 consensus_queue.cc:260] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.103176 7652 ts_tablet_manager.cc:1398] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:34.103168 7651 raft_consensus.cc:348] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.103258 7651 raft_consensus.cc:374] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.103307 7651 raft_consensus.cc:729] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.103435 7651 consensus_queue.cc:260] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.103688 7651 ts_tablet_manager.cc:1398] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:34.104152 7650 tablet_bootstrap.cc:492] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b: No bootstrap required, opened a new log
I20240503 13:59:34.104233 7650 ts_tablet_manager.cc:1367] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:34.104740 7650 raft_consensus.cc:348] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.104820 7650 raft_consensus.cc:374] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.104846 7650 raft_consensus.cc:729] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.104938 7650 consensus_queue.cc:260] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.105165 7650 ts_tablet_manager.cc:1398] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:34.115861 7657 raft_consensus.cc:482] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.115967 7657 raft_consensus.cc:504] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.116621 7657 leader_election.cc:288] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:34.119738 6909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527c7e7cf4f24ce3bed276c1860aa041" candidate_uuid: "791cddb67b7e415aad7edb5151037418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a58bb8fe574f5ab1485c59bf994570" is_pre_election: true
I20240503 13:59:34.119938 6909 raft_consensus.cc:2454] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791cddb67b7e415aad7edb5151037418 in term 0.
I20240503 13:59:34.120271 6711 leader_election.cc:302] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52a58bb8fe574f5ab1485c59bf994570, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:34.120461 7657 raft_consensus.cc:2781] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.120522 7657 raft_consensus.cc:482] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.120568 7657 raft_consensus.cc:3037] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.121279 7657 raft_consensus.cc:504] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.121526 7657 leader_election.cc:288] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 election: Requested vote from peers 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:34.121802 6909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527c7e7cf4f24ce3bed276c1860aa041" candidate_uuid: "791cddb67b7e415aad7edb5151037418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a58bb8fe574f5ab1485c59bf994570"
I20240503 13:59:34.121892 6909 raft_consensus.cc:3037] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.122632 6909 raft_consensus.cc:2454] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791cddb67b7e415aad7edb5151037418 in term 1.
I20240503 13:59:34.122879 6711 leader_election.cc:302] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52a58bb8fe574f5ab1485c59bf994570, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:34.123085 7657 raft_consensus.cc:2781] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.123307 7657 raft_consensus.cc:686] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Becoming Leader. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Running, Role: LEADER
I20240503 13:59:34.123557 7657 consensus_queue.cc:237] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.125690 6632 catalog_manager.cc:5484] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 reported cstate change: term changed from 0 to 1, leader changed from <none> to 791cddb67b7e415aad7edb5151037418 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "791cddb67b7e415aad7edb5151037418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } }
W20240503 13:59:34.131731 6825 tablet.cc:2369] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:34.147903 7659 raft_consensus.cc:482] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.147997 7659 raft_consensus.cc:504] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.148785 7659 leader_election.cc:288] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:34.151970 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6409f411c5d24e97b34aa9cce7c5a5ea" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c195eab0b34945afd10f3291ed080b" is_pre_election: true
I20240503 13:59:34.152170 7040 raft_consensus.cc:2454] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 0.
I20240503 13:59:34.152489 6842 leader_election.cc:302] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 52a58bb8fe574f5ab1485c59bf994570; no voters:
I20240503 13:59:34.152675 7659 raft_consensus.cc:2781] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.152732 7659 raft_consensus.cc:482] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.152756 7659 raft_consensus.cc:3037] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.153507 7659 raft_consensus.cc:504] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.153771 7659 leader_election.cc:288] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Requested vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:34.154100 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6409f411c5d24e97b34aa9cce7c5a5ea" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c195eab0b34945afd10f3291ed080b"
I20240503 13:59:34.154219 7040 raft_consensus.cc:3037] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.155249 7040 raft_consensus.cc:2454] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 1.
I20240503 13:59:34.155517 6842 leader_election.cc:302] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 52a58bb8fe574f5ab1485c59bf994570; no voters:
I20240503 13:59:34.155663 7659 raft_consensus.cc:2781] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.155874 7659 raft_consensus.cc:686] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Becoming Leader. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Running, Role: LEADER
I20240503 13:59:34.156144 7659 consensus_queue.cc:237] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.158737 6632 catalog_manager.cc:5484] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } }
I20240503 13:59:34.213618 7657 raft_consensus.cc:482] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.213735 7657 raft_consensus.cc:504] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.214147 7657 leader_election.cc:288] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
W20240503 13:59:34.216516 7087 tablet.cc:2369] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:34.217499 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a871746624e442597588274959d5e24" candidate_uuid: "791cddb67b7e415aad7edb5151037418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c195eab0b34945afd10f3291ed080b" is_pre_election: true
I20240503 13:59:34.217613 7040 raft_consensus.cc:2454] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791cddb67b7e415aad7edb5151037418 in term 0.
I20240503 13:59:34.217840 6711 leader_election.cc:302] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:34.217986 7657 raft_consensus.cc:2781] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.218039 7657 raft_consensus.cc:482] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.218070 7657 raft_consensus.cc:3037] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.218842 7657 raft_consensus.cc:504] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.219076 7657 leader_election.cc:288] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 election: Requested vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:34.219388 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a871746624e442597588274959d5e24" candidate_uuid: "791cddb67b7e415aad7edb5151037418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c195eab0b34945afd10f3291ed080b"
I20240503 13:59:34.219486 7040 raft_consensus.cc:3037] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.220185 7040 raft_consensus.cc:2454] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791cddb67b7e415aad7edb5151037418 in term 1.
I20240503 13:59:34.220425 6711 leader_election.cc:302] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:34.220600 7657 raft_consensus.cc:2781] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.220708 7657 raft_consensus.cc:686] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Becoming Leader. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Running, Role: LEADER
I20240503 13:59:34.220865 7657 consensus_queue.cc:237] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.222543 6632 catalog_manager.cc:5484] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 reported cstate change: term changed from 0 to 1, leader changed from <none> to 791cddb67b7e415aad7edb5151037418 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "791cddb67b7e415aad7edb5151037418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } }
W20240503 13:59:34.296730 6956 tablet.cc:2369] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:59:34.358062 7669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:34.358273 7669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:34.360967 7669 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 13:59:34.442471 6909 raft_consensus.cc:1261] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Refusing update from remote peer 791cddb67b7e415aad7edb5151037418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.443549 7040 raft_consensus.cc:1261] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Refusing update from remote peer 791cddb67b7e415aad7edb5151037418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.443922 7657 consensus_queue.cc:1035] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.444295 7657 consensus_queue.cc:1035] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.445561 7040 raft_consensus.cc:1261] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.446004 7666 consensus_queue.cc:1035] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.450501 7699 mvcc.cc:205] Tried to move back new op lower bound from 7023594596113641472 to 7023594595209601024. Current Snapshot: MvccSnapshot[applied={T|T < 7023594596113641472}]
I20240503 13:59:34.456360 7700 mvcc.cc:205] Tried to move back new op lower bound from 7023594596110618624 to 7023594594811555840. Current Snapshot: MvccSnapshot[applied={T|T < 7023594596110618624}]
I20240503 13:59:34.467968 7704 mvcc.cc:205] Tried to move back new op lower bound from 7023594596113641472 to 7023594595209601024. Current Snapshot: MvccSnapshot[applied={T|T < 7023594596113641472}]
INSERT report
rows total: 2000
time total: 90.4563 ms
time per row: 0.0452281 ms
I20240503 13:59:34.490355 6633 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60082:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:34.495146 6757 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9724b166dc4ca690fc912049779985 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb88829318084b1e8ec739465823de3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.495224 6889 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9724b166dc4ca690fc912049779985 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb88829318084b1e8ec739465823de3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.495465 6757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9724b166dc4ca690fc912049779985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.495489 6889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9724b166dc4ca690fc912049779985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.496925 7651 tablet_bootstrap.cc:492] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418: Bootstrap starting.
I20240503 13:59:34.497632 7019 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb5bbc56575455eb122d1e401fbdc64 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb88829318084b1e8ec739465823de3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.497717 7651 tablet_bootstrap.cc:654] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.497696 7020 tablet_service.cc:1467] Processing CreateTablet for tablet f2b880960ed44b20b81ddf91c8534638 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb88829318084b1e8ec739465823de3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.497867 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb5bbc56575455eb122d1e401fbdc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.497999 7020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b880960ed44b20b81ddf91c8534638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.498243 7652 tablet_bootstrap.cc:492] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: Bootstrap starting.
I20240503 13:59:34.498220 6889 tablet_service.cc:1467] Processing CreateTablet for tablet f2b880960ed44b20b81ddf91c8534638 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb88829318084b1e8ec739465823de3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.498389 6889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b880960ed44b20b81ddf91c8534638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.498940 7652 tablet_bootstrap.cc:654] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.499522 6757 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb5bbc56575455eb122d1e401fbdc64 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bb88829318084b1e8ec739465823de3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:34.499713 6757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb5bbc56575455eb122d1e401fbdc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:34.499915 7650 tablet_bootstrap.cc:492] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b: Bootstrap starting.
I20240503 13:59:34.500180 7651 tablet_bootstrap.cc:492] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418: No bootstrap required, opened a new log
I20240503 13:59:34.500247 7651 ts_tablet_manager.cc:1367] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:34.500532 7650 tablet_bootstrap.cc:654] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.500691 7651 raft_consensus.cc:348] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:34.500773 7651 raft_consensus.cc:374] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.500800 7651 raft_consensus.cc:729] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.500912 7651 consensus_queue.cc:260] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:34.501178 7651 ts_tablet_manager.cc:1398] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:34.501457 7651 tablet_bootstrap.cc:492] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418: Bootstrap starting.
I20240503 13:59:34.501963 7650 tablet_bootstrap.cc:492] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b: No bootstrap required, opened a new log
I20240503 13:59:34.502050 7650 ts_tablet_manager.cc:1367] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:34.502562 7651 tablet_bootstrap.cc:654] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.502504 7650 raft_consensus.cc:348] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.502617 7650 raft_consensus.cc:374] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.502652 7650 raft_consensus.cc:729] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.502779 7652 tablet_bootstrap.cc:492] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: No bootstrap required, opened a new log
I20240503 13:59:34.502760 7650 consensus_queue.cc:260] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.502851 7652 ts_tablet_manager.cc:1367] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 13:59:34.503044 7650 ts_tablet_manager.cc:1398] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.503268 7650 tablet_bootstrap.cc:492] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b: Bootstrap starting.
I20240503 13:59:34.503388 7652 raft_consensus.cc:348] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:34.503479 7652 raft_consensus.cc:374] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.503504 7652 raft_consensus.cc:729] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.503605 7652 consensus_queue.cc:260] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:34.503849 7652 ts_tablet_manager.cc:1398] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.503973 7652 tablet_bootstrap.cc:492] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570: Bootstrap starting.
I20240503 13:59:34.504103 7650 tablet_bootstrap.cc:654] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.504721 7652 tablet_bootstrap.cc:654] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:34.505757 7650 tablet_bootstrap.cc:492] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b: No bootstrap required, opened a new log
I20240503 13:59:34.505820 7651 tablet_bootstrap.cc:492] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418: No bootstrap required, opened a new log
I20240503 13:59:34.505846 7650 ts_tablet_manager.cc:1367] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:34.505894 7651 ts_tablet_manager.cc:1367] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 13:59:34.506234 7650 raft_consensus.cc:348] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.506342 7650 raft_consensus.cc:374] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.506296 7651 raft_consensus.cc:348] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.506376 7650 raft_consensus.cc:729] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.506404 7651 raft_consensus.cc:374] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.506443 7651 raft_consensus.cc:729] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.506492 7650 consensus_queue.cc:260] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.506546 7651 consensus_queue.cc:260] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.506762 7650 ts_tablet_manager.cc:1398] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:34.507803 7652 tablet_bootstrap.cc:492] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570: No bootstrap required, opened a new log
I20240503 13:59:34.507880 7652 ts_tablet_manager.cc:1367] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:34.508188 7651 ts_tablet_manager.cc:1398] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:34.508255 7652 raft_consensus.cc:348] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.508342 7652 raft_consensus.cc:374] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:34.508373 7652 raft_consensus.cc:729] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Initialized, Role: FOLLOWER
I20240503 13:59:34.508468 7652 consensus_queue.cc:260] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.508739 7652 ts_tablet_manager.cc:1398] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:34.610095 7666 raft_consensus.cc:482] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.610239 7666 raft_consensus.cc:504] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:34.610642 7666 leader_election.cc:288] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:34.613720 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9724b166dc4ca690fc912049779985" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418" is_pre_election: true
I20240503 13:59:34.613932 6778 raft_consensus.cc:2454] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 0.
I20240503 13:59:34.614264 6842 leader_election.cc:302] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52a58bb8fe574f5ab1485c59bf994570, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:34.614464 7666 raft_consensus.cc:2781] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.614531 7666 raft_consensus.cc:482] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.614560 7666 raft_consensus.cc:3037] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.615391 7666 raft_consensus.cc:504] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:34.615649 7666 leader_election.cc:288] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Requested vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:34.615986 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9724b166dc4ca690fc912049779985" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418"
I20240503 13:59:34.616084 6778 raft_consensus.cc:3037] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.616819 6778 raft_consensus.cc:2454] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 1.
I20240503 13:59:34.617072 6842 leader_election.cc:302] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52a58bb8fe574f5ab1485c59bf994570, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:34.617210 7666 raft_consensus.cc:2781] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.617337 7666 raft_consensus.cc:686] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Becoming Leader. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Running, Role: LEADER
I20240503 13:59:34.617483 7666 consensus_queue.cc:237] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:34.618932 6636 catalog_manager.cc:5484] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } }
I20240503 13:59:34.651386 7663 raft_consensus.cc:482] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.651502 7663 raft_consensus.cc:504] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.651794 7663 leader_election.cc:288] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:34.652211 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb5bbc56575455eb122d1e401fbdc64" candidate_uuid: "791cddb67b7e415aad7edb5151037418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c195eab0b34945afd10f3291ed080b" is_pre_election: true
I20240503 13:59:34.652333 7040 raft_consensus.cc:2454] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791cddb67b7e415aad7edb5151037418 in term 0.
I20240503 13:59:34.652637 6711 leader_election.cc:302] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:34.652822 7663 raft_consensus.cc:2781] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.652904 7663 raft_consensus.cc:482] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.652956 7663 raft_consensus.cc:3037] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.653970 7663 raft_consensus.cc:504] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.654265 7663 leader_election.cc:288] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 election: Requested vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:34.654675 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb5bbc56575455eb122d1e401fbdc64" candidate_uuid: "791cddb67b7e415aad7edb5151037418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c195eab0b34945afd10f3291ed080b"
I20240503 13:59:34.654778 7040 raft_consensus.cc:3037] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.655640 7040 raft_consensus.cc:2454] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791cddb67b7e415aad7edb5151037418 in term 1.
I20240503 13:59:34.655923 6711 leader_election.cc:302] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:34.656078 7663 raft_consensus.cc:2781] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.656157 7663 raft_consensus.cc:686] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Becoming Leader. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Running, Role: LEADER
I20240503 13:59:34.656291 7663 consensus_queue.cc:237] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:34.657871 6636 catalog_manager.cc:5484] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 reported cstate change: term changed from 0 to 1, leader changed from <none> to 791cddb67b7e415aad7edb5151037418 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "791cddb67b7e415aad7edb5151037418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } }
I20240503 13:59:34.725342 7666 raft_consensus.cc:482] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:34.725499 7666 raft_consensus.cc:504] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.725860 7666 leader_election.cc:288] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:34.726305 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b880960ed44b20b81ddf91c8534638" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c195eab0b34945afd10f3291ed080b" is_pre_election: true
I20240503 13:59:34.726430 7040 raft_consensus.cc:2454] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 0.
I20240503 13:59:34.726737 6842 leader_election.cc:302] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 52a58bb8fe574f5ab1485c59bf994570; no voters:
I20240503 13:59:34.726899 7666 raft_consensus.cc:2781] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:34.726958 7666 raft_consensus.cc:482] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:34.726992 7666 raft_consensus.cc:3037] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.727861 7666 raft_consensus.cc:504] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.728175 7666 leader_election.cc:288] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Requested vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:34.728579 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b880960ed44b20b81ddf91c8534638" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c195eab0b34945afd10f3291ed080b"
I20240503 13:59:34.728706 7040 raft_consensus.cc:3037] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:34.729753 7040 raft_consensus.cc:2454] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 1.
I20240503 13:59:34.730062 6842 leader_election.cc:302] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 52a58bb8fe574f5ab1485c59bf994570; no voters:
I20240503 13:59:34.730223 7666 raft_consensus.cc:2781] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:34.730329 7666 raft_consensus.cc:686] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Becoming Leader. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Running, Role: LEADER
I20240503 13:59:34.730486 7666 consensus_queue.cc:237] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:34.732326 6636 catalog_manager.cc:5484] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } }
W20240503 13:59:34.890416 7725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:34.890676 7725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:34.893747 7725 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 13:59:34.981467 6778 raft_consensus.cc:1261] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.981467 7040 raft_consensus.cc:1261] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.981848 7659 consensus_queue.cc:1035] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.982244 7659 consensus_queue.cc:1035] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:34.984325 7039 raft_consensus.cc:1261] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Refusing update from remote peer 791cddb67b7e415aad7edb5151037418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:34.985352 7663 consensus_queue.cc:1035] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 83.8716 ms
time per row: 0.0419358 ms
I20240503 13:59:35.026041 6633 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60082:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:35.030985 6757 tablet_service.cc:1467] Processing CreateTablet for tablet a82b94a4baa84d778cc280d58c18af5b (DEFAULT_TABLE table=rebalance_test_table_2 [id=04d1a2edbd1a48bc84eae4ce2c7379ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.031328 6757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82b94a4baa84d778cc280d58c18af5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.031275 7019 tablet_service.cc:1467] Processing CreateTablet for tablet 9a52cb95109045d2bbfa3def26848224 (DEFAULT_TABLE table=rebalance_test_table_2 [id=04d1a2edbd1a48bc84eae4ce2c7379ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.031504 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a52cb95109045d2bbfa3def26848224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.031618 7020 tablet_service.cc:1467] Processing CreateTablet for tablet ef2d39830da24221a3dc44da57c0a6e8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=04d1a2edbd1a48bc84eae4ce2c7379ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.031843 7020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2d39830da24221a3dc44da57c0a6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.031890 6888 tablet_service.cc:1467] Processing CreateTablet for tablet ef2d39830da24221a3dc44da57c0a6e8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=04d1a2edbd1a48bc84eae4ce2c7379ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.032124 6888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2d39830da24221a3dc44da57c0a6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.033633 6889 tablet_service.cc:1467] Processing CreateTablet for tablet a82b94a4baa84d778cc280d58c18af5b (DEFAULT_TABLE table=rebalance_test_table_2 [id=04d1a2edbd1a48bc84eae4ce2c7379ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.033865 6889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82b94a4baa84d778cc280d58c18af5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.034194 7772 tablet_bootstrap.cc:492] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570: Bootstrap starting.
I20240503 13:59:35.034271 7771 tablet_bootstrap.cc:492] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418: Bootstrap starting.
I20240503 13:59:35.035068 7771 tablet_bootstrap.cc:654] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.035068 7772 tablet_bootstrap.cc:654] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.035502 6758 tablet_service.cc:1467] Processing CreateTablet for tablet 9a52cb95109045d2bbfa3def26848224 (DEFAULT_TABLE table=rebalance_test_table_2 [id=04d1a2edbd1a48bc84eae4ce2c7379ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.035706 6758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a52cb95109045d2bbfa3def26848224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.036113 7771 tablet_bootstrap.cc:492] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418: No bootstrap required, opened a new log
I20240503 13:59:35.036186 7771 ts_tablet_manager.cc:1367] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:35.036375 7772 tablet_bootstrap.cc:492] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570: No bootstrap required, opened a new log
I20240503 13:59:35.036438 7772 ts_tablet_manager.cc:1367] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.036614 7771 raft_consensus.cc:348] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.036697 7771 raft_consensus.cc:374] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.036725 7771 raft_consensus.cc:729] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.036842 7772 raft_consensus.cc:348] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.036847 7771 consensus_queue.cc:260] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.036914 7772 raft_consensus.cc:374] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.036945 7772 raft_consensus.cc:729] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.037102 7771 ts_tablet_manager.cc:1398] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.037261 7771 tablet_bootstrap.cc:492] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418: Bootstrap starting.
I20240503 13:59:35.037446 7772 consensus_queue.cc:260] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.037675 7770 tablet_bootstrap.cc:492] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b: Bootstrap starting.
I20240503 13:59:35.037768 7772 ts_tablet_manager.cc:1398] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.037961 7772 tablet_bootstrap.cc:492] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570: Bootstrap starting.
I20240503 13:59:35.038504 7770 tablet_bootstrap.cc:654] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.038738 7772 tablet_bootstrap.cc:654] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.039522 7770 tablet_bootstrap.cc:492] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b: No bootstrap required, opened a new log
I20240503 13:59:35.039599 7770 ts_tablet_manager.cc:1367] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.039770 7772 tablet_bootstrap.cc:492] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570: No bootstrap required, opened a new log
I20240503 13:59:35.039841 7772 ts_tablet_manager.cc:1367] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.040050 7770 raft_consensus.cc:348] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:35.040133 7770 raft_consensus.cc:374] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.040170 7770 raft_consensus.cc:729] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.040275 7770 consensus_queue.cc:260] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:35.040282 7772 raft_consensus.cc:348] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.040362 7772 raft_consensus.cc:374] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.040395 7772 raft_consensus.cc:729] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.040527 7770 ts_tablet_manager.cc:1398] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.040499 7772 consensus_queue.cc:260] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.040694 7770 tablet_bootstrap.cc:492] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b: Bootstrap starting.
I20240503 13:59:35.040743 7772 ts_tablet_manager.cc:1398] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.041570 7770 tablet_bootstrap.cc:654] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.042460 7771 tablet_bootstrap.cc:654] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.043073 7770 tablet_bootstrap.cc:492] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b: No bootstrap required, opened a new log
I20240503 13:59:35.043187 7770 ts_tablet_manager.cc:1367] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:35.043586 7770 raft_consensus.cc:348] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.043668 7770 raft_consensus.cc:374] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.043702 7770 raft_consensus.cc:729] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.043773 7771 tablet_bootstrap.cc:492] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418: No bootstrap required, opened a new log
I20240503 13:59:35.043861 7771 ts_tablet_manager.cc:1367] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 13:59:35.043833 7770 consensus_queue.cc:260] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.044100 7663 raft_consensus.cc:482] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.044160 7770 ts_tablet_manager.cc:1398] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:35.044173 7663 raft_consensus.cc:504] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.044278 7771 raft_consensus.cc:348] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:35.044348 7771 raft_consensus.cc:374] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.044375 7771 raft_consensus.cc:729] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.044448 7663 leader_election.cc:288] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:35.044499 7771 consensus_queue.cc:260] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:35.044747 7771 ts_tablet_manager.cc:1398] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:35.044791 6909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82b94a4baa84d778cc280d58c18af5b" candidate_uuid: "791cddb67b7e415aad7edb5151037418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a58bb8fe574f5ab1485c59bf994570" is_pre_election: true
I20240503 13:59:35.044904 6909 raft_consensus.cc:2454] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791cddb67b7e415aad7edb5151037418 in term 0.
I20240503 13:59:35.045192 6711 leader_election.cc:302] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52a58bb8fe574f5ab1485c59bf994570, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:35.045336 7663 raft_consensus.cc:2781] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.045385 7663 raft_consensus.cc:482] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.045430 7663 raft_consensus.cc:3037] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.046097 7663 raft_consensus.cc:504] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.046317 7663 leader_election.cc:288] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 election: Requested vote from peers 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:35.046612 6909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82b94a4baa84d778cc280d58c18af5b" candidate_uuid: "791cddb67b7e415aad7edb5151037418" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a58bb8fe574f5ab1485c59bf994570"
I20240503 13:59:35.046705 6909 raft_consensus.cc:3037] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.047545 6909 raft_consensus.cc:2454] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791cddb67b7e415aad7edb5151037418 in term 1.
I20240503 13:59:35.047770 6711 leader_election.cc:302] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52a58bb8fe574f5ab1485c59bf994570, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:35.047910 7663 raft_consensus.cc:2781] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.047986 7663 raft_consensus.cc:686] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Becoming Leader. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Running, Role: LEADER
I20240503 13:59:35.048115 7663 consensus_queue.cc:237] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.049222 6634 catalog_manager.cc:5484] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 reported cstate change: term changed from 0 to 1, leader changed from <none> to 791cddb67b7e415aad7edb5151037418 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "791cddb67b7e415aad7edb5151037418" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } }
I20240503 13:59:35.079032 7659 raft_consensus.cc:482] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.079120 7659 raft_consensus.cc:504] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.079416 7659 leader_election.cc:288] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:35.079758 7039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2d39830da24221a3dc44da57c0a6e8" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c195eab0b34945afd10f3291ed080b" is_pre_election: true
I20240503 13:59:35.079874 7039 raft_consensus.cc:2454] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 0.
I20240503 13:59:35.080124 6842 leader_election.cc:302] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 52a58bb8fe574f5ab1485c59bf994570; no voters:
I20240503 13:59:35.080268 7659 raft_consensus.cc:2781] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.080322 7659 raft_consensus.cc:482] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.080356 7659 raft_consensus.cc:3037] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.081158 7659 raft_consensus.cc:504] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.081360 7659 leader_election.cc:288] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Requested vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:35.081656 7039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2d39830da24221a3dc44da57c0a6e8" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c195eab0b34945afd10f3291ed080b"
I20240503 13:59:35.081748 7039 raft_consensus.cc:3037] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.082412 7039 raft_consensus.cc:2454] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 1.
I20240503 13:59:35.082659 6842 leader_election.cc:302] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 52a58bb8fe574f5ab1485c59bf994570; no voters:
I20240503 13:59:35.082828 7659 raft_consensus.cc:2781] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.082911 7659 raft_consensus.cc:686] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Becoming Leader. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Running, Role: LEADER
I20240503 13:59:35.083031 7659 consensus_queue.cc:237] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.084468 6634 catalog_manager.cc:5484] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } }
I20240503 13:59:35.135401 7656 raft_consensus.cc:482] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.135521 7656 raft_consensus.cc:504] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:35.136425 7656 leader_election.cc:288] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:35.140136 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a52cb95109045d2bbfa3def26848224" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418" is_pre_election: true
I20240503 13:59:35.140277 6778 raft_consensus.cc:2454] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 0.
I20240503 13:59:35.140604 6973 leader_election.cc:302] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:35.140848 7656 raft_consensus.cc:2781] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.140904 7656 raft_consensus.cc:482] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.140938 7656 raft_consensus.cc:3037] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.141760 7656 raft_consensus.cc:504] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:35.142011 7656 leader_election.cc:288] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 election: Requested vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:35.142326 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a52cb95109045d2bbfa3def26848224" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418"
I20240503 13:59:35.142429 6778 raft_consensus.cc:3037] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.143375 6778 raft_consensus.cc:2454] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 1.
I20240503 13:59:35.143643 6973 leader_election.cc:302] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:35.143839 7656 raft_consensus.cc:2781] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.144032 7656 raft_consensus.cc:686] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Becoming Leader. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Running, Role: LEADER
I20240503 13:59:35.144308 7656 consensus_queue.cc:237] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:35.146071 6634 catalog_manager.cc:5484] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b reported cstate change: term changed from 0 to 1, leader changed from <none> to 14c195eab0b34945afd10f3291ed080b (127.0.99.131). New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } }
W20240503 13:59:35.259543 7784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:35.259769 7784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:35.262501 7784 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 13:59:35.353044 6909 raft_consensus.cc:1261] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Refusing update from remote peer 791cddb67b7e415aad7edb5151037418: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.353073 6778 raft_consensus.cc:1261] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.353477 7663 consensus_queue.cc:1035] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.353650 7656 consensus_queue.cc:1035] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:35.359378 7039 raft_consensus.cc:1261] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:35.359894 7659 consensus_queue.cc:1035] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 96.0642 ms
time per row: 0.0480321 ms
I20240503 13:59:35.397362 6634 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60082:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:35.402226 6758 tablet_service.cc:1467] Processing CreateTablet for tablet 4f882dbedd134f2998642ac747567db0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bffd930039c544a68647ac9a405cf590]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.402271 6889 tablet_service.cc:1467] Processing CreateTablet for tablet 4f882dbedd134f2998642ac747567db0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bffd930039c544a68647ac9a405cf590]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.402523 6758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f882dbedd134f2998642ac747567db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.402523 6889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f882dbedd134f2998642ac747567db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.402626 6757 tablet_service.cc:1467] Processing CreateTablet for tablet 57ea09e3f1f74d74839ddb0afd90acc2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bffd930039c544a68647ac9a405cf590]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.402892 6757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ea09e3f1f74d74839ddb0afd90acc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.402943 7019 tablet_service.cc:1467] Processing CreateTablet for tablet e026fe8a768941ce84378f23b3dabff2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bffd930039c544a68647ac9a405cf590]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.403178 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e026fe8a768941ce84378f23b3dabff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.404717 7771 tablet_bootstrap.cc:492] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418: Bootstrap starting.
I20240503 13:59:35.404796 7020 tablet_service.cc:1467] Processing CreateTablet for tablet 57ea09e3f1f74d74839ddb0afd90acc2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bffd930039c544a68647ac9a405cf590]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.405004 7020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ea09e3f1f74d74839ddb0afd90acc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.405220 7770 tablet_bootstrap.cc:492] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b: Bootstrap starting.
I20240503 13:59:35.405540 7771 tablet_bootstrap.cc:654] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.405912 7770 tablet_bootstrap.cc:654] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.406188 7772 tablet_bootstrap.cc:492] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570: Bootstrap starting.
I20240503 13:59:35.407009 7770 tablet_bootstrap.cc:492] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b: No bootstrap required, opened a new log
I20240503 13:59:35.407081 7772 tablet_bootstrap.cc:654] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.407099 7770 ts_tablet_manager.cc:1367] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:35.407451 7770 raft_consensus.cc:348] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.407529 7770 raft_consensus.cc:374] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.407548 7770 raft_consensus.cc:729] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.407646 7770 consensus_queue.cc:260] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.407908 7770 ts_tablet_manager.cc:1398] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.407981 6889 tablet_service.cc:1467] Processing CreateTablet for tablet e026fe8a768941ce84378f23b3dabff2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bffd930039c544a68647ac9a405cf590]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:35.408052 7770 tablet_bootstrap.cc:492] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: Bootstrap starting.
I20240503 13:59:35.408067 7772 tablet_bootstrap.cc:492] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570: No bootstrap required, opened a new log
I20240503 13:59:35.408135 7772 ts_tablet_manager.cc:1367] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:35.408181 6889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e026fe8a768941ce84378f23b3dabff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:35.408764 7771 tablet_bootstrap.cc:492] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418: No bootstrap required, opened a new log
I20240503 13:59:35.408727 7772 raft_consensus.cc:348] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.408820 7772 raft_consensus.cc:374] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.408833 7771 ts_tablet_manager.cc:1367] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:59:35.408840 7770 tablet_bootstrap.cc:654] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.408854 7772 raft_consensus.cc:729] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.408963 7772 consensus_queue.cc:260] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.409258 7772 ts_tablet_manager.cc:1398] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:35.409374 7771 raft_consensus.cc:348] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.409456 7771 raft_consensus.cc:374] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.409489 7771 raft_consensus.cc:729] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.409615 7771 consensus_queue.cc:260] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.409866 7771 ts_tablet_manager.cc:1398] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:35.410033 7771 tablet_bootstrap.cc:492] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418: Bootstrap starting.
I20240503 13:59:35.410149 7772 tablet_bootstrap.cc:492] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570: Bootstrap starting.
I20240503 13:59:35.410625 7770 tablet_bootstrap.cc:492] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: No bootstrap required, opened a new log
I20240503 13:59:35.410693 7770 ts_tablet_manager.cc:1367] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:35.410998 7772 tablet_bootstrap.cc:654] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.411127 7770 raft_consensus.cc:348] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.411211 7770 raft_consensus.cc:374] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.411259 7770 raft_consensus.cc:729] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.411367 7770 consensus_queue.cc:260] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.411612 7770 ts_tablet_manager.cc:1398] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:35.412348 7771 tablet_bootstrap.cc:654] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:35.413005 7772 tablet_bootstrap.cc:492] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570: No bootstrap required, opened a new log
I20240503 13:59:35.413133 7772 ts_tablet_manager.cc:1367] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:35.413632 7772 raft_consensus.cc:348] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.413808 7772 raft_consensus.cc:374] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.413913 7771 tablet_bootstrap.cc:492] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418: No bootstrap required, opened a new log
I20240503 13:59:35.413997 7771 ts_tablet_manager.cc:1367] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 13:59:35.414444 7772 raft_consensus.cc:729] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.414424 7771 raft_consensus.cc:348] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.414513 7771 raft_consensus.cc:374] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:35.414559 7771 raft_consensus.cc:729] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Initialized, Role: FOLLOWER
I20240503 13:59:35.414637 7772 consensus_queue.cc:260] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.414682 7771 consensus_queue.cc:260] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.414954 7771 ts_tablet_manager.cc:1398] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:35.414974 7772 ts_tablet_manager.cc:1398] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:35.632196 7819 raft_consensus.cc:482] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.632342 7819 raft_consensus.cc:504] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.632782 7819 leader_election.cc:288] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:35.635874 6909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e026fe8a768941ce84378f23b3dabff2" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a58bb8fe574f5ab1485c59bf994570" is_pre_election: true
I20240503 13:59:35.636008 6909 raft_consensus.cc:2454] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 0.
I20240503 13:59:35.636298 6973 leader_election.cc:302] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 52a58bb8fe574f5ab1485c59bf994570; no voters:
I20240503 13:59:35.636473 7819 raft_consensus.cc:2781] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.636538 7819 raft_consensus.cc:482] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.636590 7819 raft_consensus.cc:3037] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.637441 7819 raft_consensus.cc:504] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.637697 7819 leader_election.cc:288] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 election: Requested vote from peers 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:35.638036 6909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e026fe8a768941ce84378f23b3dabff2" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a58bb8fe574f5ab1485c59bf994570"
I20240503 13:59:35.638139 6909 raft_consensus.cc:3037] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.638811 6909 raft_consensus.cc:2454] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 1.
I20240503 13:59:35.639029 6973 leader_election.cc:302] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 52a58bb8fe574f5ab1485c59bf994570; no voters:
I20240503 13:59:35.639163 7819 raft_consensus.cc:2781] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.639262 7819 raft_consensus.cc:686] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Becoming Leader. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Running, Role: LEADER
I20240503 13:59:35.639394 7819 consensus_queue.cc:237] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.640774 6632 catalog_manager.cc:5484] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b reported cstate change: term changed from 0 to 1, leader changed from <none> to 14c195eab0b34945afd10f3291ed080b (127.0.99.131). New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } }
I20240503 13:59:35.765170 7819 raft_consensus.cc:482] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.765306 7819 raft_consensus.cc:504] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.765668 7819 leader_election.cc:288] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:35.766182 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ea09e3f1f74d74839ddb0afd90acc2" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418" is_pre_election: true
I20240503 13:59:35.766310 6778 raft_consensus.cc:2454] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 0.
I20240503 13:59:35.766595 6973 leader_election.cc:302] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:35.766757 7819 raft_consensus.cc:2781] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.766817 7819 raft_consensus.cc:482] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.766847 7819 raft_consensus.cc:3037] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.767736 7819 raft_consensus.cc:504] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.768008 7819 leader_election.cc:288] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 election: Requested vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:35.768369 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ea09e3f1f74d74839ddb0afd90acc2" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418"
I20240503 13:59:35.768496 6778 raft_consensus.cc:3037] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.769351 6778 raft_consensus.cc:2454] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 1.
I20240503 13:59:35.769606 6973 leader_election.cc:302] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:35.769788 7819 raft_consensus.cc:2781] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.769882 7819 raft_consensus.cc:686] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Becoming Leader. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Running, Role: LEADER
I20240503 13:59:35.770016 7819 consensus_queue.cc:237] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } }
I20240503 13:59:35.771450 6632 catalog_manager.cc:5484] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b reported cstate change: term changed from 0 to 1, leader changed from <none> to 14c195eab0b34945afd10f3291ed080b (127.0.99.131). New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } }
I20240503 13:59:35.829962 7659 raft_consensus.cc:482] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:35.830108 7659 raft_consensus.cc:504] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.830463 7659 leader_election.cc:288] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:35.830956 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f882dbedd134f2998642ac747567db0" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418" is_pre_election: true
I20240503 13:59:35.831096 6778 raft_consensus.cc:2454] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 0.
I20240503 13:59:35.831425 6842 leader_election.cc:302] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52a58bb8fe574f5ab1485c59bf994570, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:35.831641 7659 raft_consensus.cc:2781] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:35.831701 7659 raft_consensus.cc:482] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:35.831730 7659 raft_consensus.cc:3037] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.832516 7659 raft_consensus.cc:504] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.832756 7659 leader_election.cc:288] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Requested vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:35.833109 6778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f882dbedd134f2998642ac747567db0" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418"
I20240503 13:59:35.833213 6778 raft_consensus.cc:3037] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:35.833963 6778 raft_consensus.cc:2454] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 1.
I20240503 13:59:35.834228 6842 leader_election.cc:302] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52a58bb8fe574f5ab1485c59bf994570, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:35.834417 7659 raft_consensus.cc:2781] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:35.834508 7659 raft_consensus.cc:686] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Becoming Leader. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Running, Role: LEADER
I20240503 13:59:35.834667 7659 consensus_queue.cc:237] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:35.836177 6632 catalog_manager.cc:5484] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } }
W20240503 13:59:35.960892 7830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:35.961119 7830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:35.964126 7830 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 13:59:36.042023 6778 raft_consensus.cc:1261] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.042469 7659 consensus_queue.cc:1035] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.045023 6777 raft_consensus.cc:1261] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.045131 6909 raft_consensus.cc:1261] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.045651 7656 consensus_queue.cc:1035] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.045672 7819 consensus_queue.cc:1035] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 76.567 ms
time per row: 0.0382835 ms
I20240503 13:59:36.083422 6634 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60082:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 13:59:36.088444 6889 tablet_service.cc:1467] Processing CreateTablet for tablet 70583380c4e24b4694fc5477d880d944 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b4f4315b2d34ad8a7a1bdd75ec294ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.088577 7020 tablet_service.cc:1467] Processing CreateTablet for tablet d118ea0cbd824762906095329740d0a3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b4f4315b2d34ad8a7a1bdd75ec294ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.088618 6757 tablet_service.cc:1467] Processing CreateTablet for tablet 70583380c4e24b4694fc5477d880d944 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b4f4315b2d34ad8a7a1bdd75ec294ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.088778 6889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70583380c4e24b4694fc5477d880d944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.088831 7020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d118ea0cbd824762906095329740d0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.088876 6757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70583380c4e24b4694fc5477d880d944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.090621 7879 tablet_bootstrap.cc:492] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418: Bootstrap starting.
I20240503 13:59:36.090997 6758 tablet_service.cc:1467] Processing CreateTablet for tablet d118ea0cbd824762906095329740d0a3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b4f4315b2d34ad8a7a1bdd75ec294ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.091094 7881 tablet_bootstrap.cc:492] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570: Bootstrap starting.
I20240503 13:59:36.091104 7880 tablet_bootstrap.cc:492] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: Bootstrap starting.
I20240503 13:59:36.091197 6758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d118ea0cbd824762906095329740d0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.091506 7879 tablet_bootstrap.cc:654] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.091886 7881 tablet_bootstrap.cc:654] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.091886 7880 tablet_bootstrap.cc:654] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.092768 7020 tablet_service.cc:1467] Processing CreateTablet for tablet 4aec442fecd64614858bce4a5bce5db4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b4f4315b2d34ad8a7a1bdd75ec294ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.092976 7020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aec442fecd64614858bce4a5bce5db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.093345 7880 tablet_bootstrap.cc:492] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: No bootstrap required, opened a new log
I20240503 13:59:36.093345 6889 tablet_service.cc:1467] Processing CreateTablet for tablet 4aec442fecd64614858bce4a5bce5db4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1b4f4315b2d34ad8a7a1bdd75ec294ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 13:59:36.093441 7880 ts_tablet_manager.cc:1367] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:36.093575 6889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aec442fecd64614858bce4a5bce5db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:36.094139 7881 tablet_bootstrap.cc:492] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570: No bootstrap required, opened a new log
I20240503 13:59:36.094202 7880 raft_consensus.cc:348] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.094267 7881 ts_tablet_manager.cc:1367] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:36.094362 7880 raft_consensus.cc:374] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.094439 7880 raft_consensus.cc:729] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.094640 7880 consensus_queue.cc:260] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.094815 7881 raft_consensus.cc:348] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.094980 7881 raft_consensus.cc:374] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.095059 7881 raft_consensus.cc:729] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.095058 7880 ts_tablet_manager.cc:1398] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:36.095319 7880 tablet_bootstrap.cc:492] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: Bootstrap starting.
I20240503 13:59:36.096280 7879 tablet_bootstrap.cc:492] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418: No bootstrap required, opened a new log
I20240503 13:59:36.096359 7879 ts_tablet_manager.cc:1367] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:36.096333 7881 consensus_queue.cc:260] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.096679 7880 tablet_bootstrap.cc:654] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.096681 7881 ts_tablet_manager.cc:1398] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:36.096827 7881 tablet_bootstrap.cc:492] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570: Bootstrap starting.
I20240503 13:59:36.096774 7879 raft_consensus.cc:348] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.096863 7879 raft_consensus.cc:374] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.096899 7879 raft_consensus.cc:729] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.097002 7879 consensus_queue.cc:260] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.097241 7879 ts_tablet_manager.cc:1398] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.097374 7879 tablet_bootstrap.cc:492] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418: Bootstrap starting.
I20240503 13:59:36.097595 7881 tablet_bootstrap.cc:654] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.098162 7879 tablet_bootstrap.cc:654] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:36.099114 7881 tablet_bootstrap.cc:492] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570: No bootstrap required, opened a new log
I20240503 13:59:36.099263 7881 ts_tablet_manager.cc:1367] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:36.099488 7880 tablet_bootstrap.cc:492] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: No bootstrap required, opened a new log
I20240503 13:59:36.099570 7880 ts_tablet_manager.cc:1367] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:36.099771 7879 tablet_bootstrap.cc:492] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418: No bootstrap required, opened a new log
I20240503 13:59:36.099735 7881 raft_consensus.cc:348] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:36.099819 7881 raft_consensus.cc:374] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.099865 7881 raft_consensus.cc:729] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.099872 7879 ts_tablet_manager.cc:1367] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:36.099972 7880 raft_consensus.cc:348] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:36.100067 7880 raft_consensus.cc:374] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.100033 7881 consensus_queue.cc:260] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:36.100108 7880 raft_consensus.cc:729] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.100229 7880 consensus_queue.cc:260] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:36.100301 7879 raft_consensus.cc:348] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.100363 7881 ts_tablet_manager.cc:1398] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.100384 7879 raft_consensus.cc:374] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:36.100412 7879 raft_consensus.cc:729] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.100473 7880 ts_tablet_manager.cc:1398] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.100546 7879 consensus_queue.cc:260] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.100859 7879 ts_tablet_manager.cc:1398] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.161338 7659 raft_consensus.cc:482] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:36.161481 7659 raft_consensus.cc:504] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.161895 7659 leader_election.cc:288] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:36.162353 6777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70583380c4e24b4694fc5477d880d944" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418" is_pre_election: true
I20240503 13:59:36.162505 6777 raft_consensus.cc:2454] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 0.
I20240503 13:59:36.162813 6842 leader_election.cc:302] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52a58bb8fe574f5ab1485c59bf994570, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:36.163029 7659 raft_consensus.cc:2781] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:36.163093 7659 raft_consensus.cc:482] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:36.163134 7659 raft_consensus.cc:3037] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.164151 7659 raft_consensus.cc:504] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.164450 7659 leader_election.cc:288] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Requested vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:36.165026 6777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70583380c4e24b4694fc5477d880d944" candidate_uuid: "52a58bb8fe574f5ab1485c59bf994570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418"
I20240503 13:59:36.165184 6777 raft_consensus.cc:3037] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.165921 6777 raft_consensus.cc:2454] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a58bb8fe574f5ab1485c59bf994570 in term 1.
I20240503 13:59:36.166201 6842 leader_election.cc:302] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52a58bb8fe574f5ab1485c59bf994570, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:36.166389 7659 raft_consensus.cc:2781] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:36.166513 7659 raft_consensus.cc:686] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Becoming Leader. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Running, Role: LEADER
I20240503 13:59:36.166643 7659 consensus_queue.cc:237] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.168224 6634 catalog_manager.cc:5484] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } }
I20240503 13:59:36.387588 7656 raft_consensus.cc:482] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:36.387725 7656 raft_consensus.cc:504] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:36.388077 7656 leader_election.cc:288] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:36.388478 6909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aec442fecd64614858bce4a5bce5db4" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a58bb8fe574f5ab1485c59bf994570" is_pre_election: true
I20240503 13:59:36.388619 6909 raft_consensus.cc:2454] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 0.
I20240503 13:59:36.388935 6973 leader_election.cc:302] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 52a58bb8fe574f5ab1485c59bf994570; no voters:
I20240503 13:59:36.389091 7656 raft_consensus.cc:2781] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:36.389145 7656 raft_consensus.cc:482] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:36.389180 7656 raft_consensus.cc:3037] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.390053 7656 raft_consensus.cc:504] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:36.390269 7656 leader_election.cc:288] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 election: Requested vote from peers 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:36.390571 6909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aec442fecd64614858bce4a5bce5db4" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a58bb8fe574f5ab1485c59bf994570"
I20240503 13:59:36.390666 6909 raft_consensus.cc:3037] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.391448 6909 raft_consensus.cc:2454] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 1.
I20240503 13:59:36.391698 6973 leader_election.cc:302] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 52a58bb8fe574f5ab1485c59bf994570; no voters:
I20240503 13:59:36.391868 7656 raft_consensus.cc:2781] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:36.391947 7656 raft_consensus.cc:686] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Becoming Leader. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Running, Role: LEADER
I20240503 13:59:36.392094 7656 consensus_queue.cc:237] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } }
I20240503 13:59:36.393560 6634 catalog_manager.cc:5484] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b reported cstate change: term changed from 0 to 1, leader changed from <none> to 14c195eab0b34945afd10f3291ed080b (127.0.99.131). New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } }
I20240503 13:59:36.422576 7656 raft_consensus.cc:482] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:36.422670 7656 raft_consensus.cc:504] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.422943 7656 leader_election.cc:288] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:36.423393 6777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d118ea0cbd824762906095329740d0a3" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418" is_pre_election: true
I20240503 13:59:36.423516 6777 raft_consensus.cc:2454] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 0.
I20240503 13:59:36.423764 6973 leader_election.cc:302] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:36.423911 7656 raft_consensus.cc:2781] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:36.423969 7656 raft_consensus.cc:482] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:36.424003 7656 raft_consensus.cc:3037] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.424726 7656 raft_consensus.cc:504] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.424935 7656 leader_election.cc:288] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 election: Requested vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:36.425235 6777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d118ea0cbd824762906095329740d0a3" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "791cddb67b7e415aad7edb5151037418"
I20240503 13:59:36.425360 6777 raft_consensus.cc:3037] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:36.426071 6777 raft_consensus.cc:2454] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 1.
I20240503 13:59:36.426309 6973 leader_election.cc:302] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:36.426443 7656 raft_consensus.cc:2781] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:36.426518 7656 raft_consensus.cc:686] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Becoming Leader. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Running, Role: LEADER
I20240503 13:59:36.426654 7656 consensus_queue.cc:237] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } }
I20240503 13:59:36.428083 6634 catalog_manager.cc:5484] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b reported cstate change: term changed from 0 to 1, leader changed from <none> to 14c195eab0b34945afd10f3291ed080b (127.0.99.131). New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } }
W20240503 13:59:36.554539 7891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:36.554755 7891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:36.557677 7891 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 13:59:36.568317 7659 consensus_queue.cc:1035] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240503 13:59:36.625113 6777 raft_consensus.cc:1261] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.625533 7656 consensus_queue.cc:1035] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:36.631685 6909 raft_consensus.cc:1261] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:36.632542 7819 consensus_queue.cc:1035] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 71.241 ms
time per row: 0.0356205 ms
I20240503 13:59:36.667779 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:41965
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:42439
--webserver_port=44053
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:59:36.770541 7940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:36.770762 7940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:36.770817 7940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:36.773638 7940 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:36.773698 7940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:36.773810 7940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 13:59:36.777164 7940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:41965
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=44053
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.7940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:36.777942 7940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:36.779016 7940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 13:59:36.788045 7940 server_base.cc:1034] running on GCE node
W20240503 13:59:36.788825 7945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:36.789541 7948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:36.790864 7946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:36.791270 7940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:36.791769 7940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:36.792905 7940 hybrid_clock.cc:648] HybridClock initialized: now 1714744776792893 us; error 27 us; skew 500 ppm
I20240503 13:59:36.794600 7940 webserver.cc:460] Webserver started at http://127.0.99.132:44053/ using document root <none> and password file <none>
I20240503 13:59:36.795114 7940 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:36.795184 7940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:36.798192 7940 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:36.799979 7954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:36.800848 7940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:36.800956 7940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "40e8eacd02b44bb8bbec0a1dc205a671"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:36.801337 7940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:36.821322 7940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:36.821841 7940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:36.821980 7940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:36.822428 7940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:36.823484 7961 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 13:59:36.826386 7940 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 13:59:36.826432 7940 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 13:59:36.826478 7940 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 13:59:36.827839 7940 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 13:59:36.827877 7940 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:36.828070 7961 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap starting.
I20240503 13:59:36.837314 7940 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:41965
I20240503 13:59:36.837350 8068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:41965 every 8 connection(s)
I20240503 13:59:36.838148 7940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 13:59:36.840549 7961 log.cc:824] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:36.841866 7961 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:36.842161 7961 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap complete.
I20240503 13:59:36.843060 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 7940
I20240503 13:59:36.843359 7961 ts_tablet_manager.cc:1367] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.008s
I20240503 13:59:36.843550 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:39967
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:42439
--webserver_port=40171
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:59:36.846575 7961 raft_consensus.cc:348] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:36.846912 8069 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:36.846935 7961 raft_consensus.cc:729] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e8eacd02b44bb8bbec0a1dc205a671, State: Initialized, Role: FOLLOWER
I20240503 13:59:36.847122 8069 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:36.847393 7961 consensus_queue.cc:260] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:36.847597 8069 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:36.848174 7961 ts_tablet_manager.cc:1398] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240503 13:59:36.849388 6632 ts_manager.cc:194] Re-registered known tserver with Master: 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132:41965)
I20240503 13:59:36.850450 6632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:45719
I20240503 13:59:36.852102 8069 heartbeater.cc:503] Master 127.0.99.190:42439 was elected leader, sending a full tablet report...
W20240503 13:59:36.947932 8073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:36.948109 8073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:36.948151 8073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:36.951023 8073 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:36.951074 8073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:36.951160 8073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 13:59:36.954553 8073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:39967
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=40171
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.8073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:36.955394 8073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:36.956379 8073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:36.964506 8079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:36.965597 8082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:36.966390 8080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:36.967180 8073 server_base.cc:1034] running on GCE node
I20240503 13:59:36.967494 8073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:36.967993 8073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:36.969143 8073 hybrid_clock.cc:648] HybridClock initialized: now 1714744776969134 us; error 24 us; skew 500 ppm
I20240503 13:59:36.970933 8073 webserver.cc:460] Webserver started at http://127.0.99.133:40171/ using document root <none> and password file <none>
I20240503 13:59:36.971483 8073 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:36.971552 8073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:36.974823 8073 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240503 13:59:36.976527 8088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:36.977366 8073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:36.977447 8073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:36.977798 8073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:37.002701 8073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:37.003309 8073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:37.003434 8073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:37.003955 8073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:37.004835 8073 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:37.004884 8073 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.004932 8073 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:37.004953 8073 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.013087 8073 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39967
I20240503 13:59:37.013113 8201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39967 every 8 connection(s)
I20240503 13:59:37.014060 8073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 13:59:37.020984 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 8073
I20240503 13:59:37.021394 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:44579
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:42439
--webserver_port=43919
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:59:37.021764 8202 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:37.021963 8202 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:37.022357 8202 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:37.023216 6632 ts_manager.cc:194] Re-registered known tserver with Master: 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133:39967)
I20240503 13:59:37.023970 6632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:44403
W20240503 13:59:37.088912 8070 tablet.cc:2369] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 13:59:37.127379 8205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:37.127604 8205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:37.127655 8205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:37.131481 8205 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:37.131553 8205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:37.131641 8205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 13:59:37.136340 8205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:44579
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=43919
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.8205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:37.137264 8205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:37.138523 8205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:37.147783 8213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:37.148488 8211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:37.149374 8210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:37.150530 8205 server_base.cc:1034] running on GCE node
I20240503 13:59:37.150887 8205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:37.151458 8205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:37.152604 8205 hybrid_clock.cc:648] HybridClock initialized: now 1714744777152590 us; error 31 us; skew 500 ppm
I20240503 13:59:37.154306 8205 webserver.cc:460] Webserver started at http://127.0.99.134:43919/ using document root <none> and password file <none>
I20240503 13:59:37.154861 8205 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:37.154937 8205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:37.158185 8205 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:37.159976 8219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.160827 8205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:37.160915 8205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "083b1c5c79a04af899eea33ba9f4502f"
format_stamp: "Formatted at 2024-05-03 13:59:29 on dist-test-slave-snql"
I20240503 13:59:37.161271 8205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:37.174145 8205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:37.174676 8205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:37.174840 8205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:37.175346 8205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:37.176278 8226 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 13:59:37.178834 8205 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 13:59:37.178882 8205 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 13:59:37.178948 8205 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 13:59:37.180294 8205 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 13:59:37.180335 8205 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:37.180546 8226 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap starting.
I20240503 13:59:37.190222 8205 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:44579
I20240503 13:59:37.190250 8333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:44579 every 8 connection(s)
I20240503 13:59:37.191162 8205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 13:59:37.193562 8226 log.cc:824] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:37.195799 8226 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:59:37.196177 8226 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap complete.
I20240503 13:59:37.197321 8226 ts_tablet_manager.cc:1367] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20240503 13:59:37.197377 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 8205
I20240503 13:59:37.197790 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:34409
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:42439
--webserver_port=43613
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 13:59:37.199846 8334 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:37.200028 8334 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:37.200407 8334 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:37.200431 8226 raft_consensus.cc:348] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:37.200852 8226 raft_consensus.cc:729] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083b1c5c79a04af899eea33ba9f4502f, State: Initialized, Role: FOLLOWER
I20240503 13:59:37.201269 8226 consensus_queue.cc:260] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:37.202723 8226 ts_tablet_manager.cc:1398] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 13:59:37.202754 6632 ts_manager.cc:194] Re-registered known tserver with Master: 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579)
I20240503 13:59:37.203830 6632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:45307
I20240503 13:59:37.205534 8334 heartbeater.cc:503] Master 127.0.99.190:42439 was elected leader, sending a full tablet report...
W20240503 13:59:37.308013 8338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:37.308238 8338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:37.308293 8338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:37.311236 8338 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:37.311324 8338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:37.311388 8338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 13:59:37.314707 8338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:34409
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=43613
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.8338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:37.315462 8338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:37.316426 8338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:37.324486 8344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:37.325531 8347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:37.326263 8345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:37.327103 8338 server_base.cc:1034] running on GCE node
I20240503 13:59:37.327450 8338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:37.327906 8338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:37.329047 8338 hybrid_clock.cc:648] HybridClock initialized: now 1714744777329024 us; error 31 us; skew 500 ppm
I20240503 13:59:37.330826 8338 webserver.cc:460] Webserver started at http://127.0.99.135:43613/ using document root <none> and password file <none>
I20240503 13:59:37.331336 8338 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:37.331401 8338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:37.334328 8338 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:37.336138 8353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.337066 8338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:37.337173 8338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4f82135907754dd8b69d08d6b7eae847"
format_stamp: "Formatted at 2024-05-03 13:59:30 on dist-test-slave-snql"
I20240503 13:59:37.337546 8338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:37.346972 8338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:37.347497 8338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:37.347627 8338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:37.348057 8338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:37.348815 8338 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:37.348857 8338 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.348891 8338 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:37.348918 8338 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:37.357125 8338 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:34409
I20240503 13:59:37.357148 8466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:34409 every 8 connection(s)
I20240503 13:59:37.358028 8338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 13:59:37.366046 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 8338
I20240503 13:59:37.366206 398 test_util.cc:268] Using random seed: 559274686
I20240503 13:59:37.366279 8467 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:37.366451 8467 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:37.366832 8467 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:37.367817 6632 ts_manager.cc:194] Re-registered known tserver with Master: 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135:34409)
I20240503 13:59:37.368428 6632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:42185
I20240503 13:59:37.374696 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 13:59:37.374908 398 test_util.cc:268] Using random seed: 559283386
I20240503 13:59:37.382746 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 13:59:37.382891 398 test_util.cc:268] Using random seed: 559291370
I20240503 13:59:37.393728 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 13:59:37.393872 398 test_util.cc:268] Using random seed: 559302350
I20240503 13:59:37.406711 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 13:59:37.406842 398 test_util.cc:268] Using random seed: 559315321
I20240503 13:59:37.424974 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240503 13:59:37.442106 8335 tablet.cc:2369] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:37.815850 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:37.823024 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:37.824599 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:37.843470 6889 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:37.845420 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:37.868541 6758 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:37.875296 8268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.025899 8202 heartbeater.cc:503] Master 127.0.99.190:42439 was elected leader, sending a full tablet report...
I20240503 13:59:38.215132 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.221431 8268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.225944 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.247591 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:38.252102 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:38.255139 6889 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:38.262430 6758 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:38.314216 8647 raft_consensus.cc:482] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:38.315670 8647 raft_consensus.cc:504] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:38.326469 8647 leader_election.cc:288] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579)
I20240503 13:59:38.346645 8288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d845691ee64bf3a5d6c2df59842a54" candidate_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083b1c5c79a04af899eea33ba9f4502f" is_pre_election: true
I20240503 13:59:38.346971 8288 raft_consensus.cc:2396] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40e8eacd02b44bb8bbec0a1dc205a671 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240503 13:59:38.347635 7955 leader_election.cc:302] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 40e8eacd02b44bb8bbec0a1dc205a671; no voters: 083b1c5c79a04af899eea33ba9f4502f
I20240503 13:59:38.354764 8647 raft_consensus.cc:2726] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:59:38.370267 8467 heartbeater.cc:503] Master 127.0.99.190:42439 was elected leader, sending a full tablet report...
I20240503 13:59:38.434558 7037 consensus_queue.cc:237] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:38.435799 6778 raft_consensus.cc:1261] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 13:59:38.436282 8546 consensus_queue.cc:1035] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 13:59:38.439005 8546 raft_consensus.cc:2932] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:38.439502 6778 raft_consensus.cc:2932] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:38.445482 6632 catalog_manager.cc:5484] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 reported cstate change: config changed from index -1 to 27, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
W20240503 13:59:38.454959 6975 consensus_peers.cc:485] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b -> Peer 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135:34409): Couldn't send request to peer 4f82135907754dd8b69d08d6b7eae847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a52cb95109045d2bbfa3def26848224. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:38.588117 7039 consensus_queue.cc:237] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:38.591257 6778 raft_consensus.cc:1261] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240503 13:59:38.591631 8680 consensus_queue.cc:1035] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240503 13:59:38.593362 8546 raft_consensus.cc:2932] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:38.593889 6778 raft_consensus.cc:2932] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:38.596998 6635 catalog_manager.cc:5484] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 reported cstate change: config changed from index -1 to 61, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
W20240503 13:59:38.599866 6973 consensus_peers.cc:485] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b -> Peer 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133:39967): Couldn't send request to peer 4c5b7c28dd2a4920a2b4e97d1afe2384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57ea09e3f1f74d74839ddb0afd90acc2. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:38.638707 7039 consensus_queue.cc:237] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:38.642107 6908 raft_consensus.cc:1261] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240503 13:59:38.642521 8546 consensus_queue.cc:1035] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20240503 13:59:38.646293 6974 consensus_peers.cc:485] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b -> Peer 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132:41965): Couldn't send request to peer 40e8eacd02b44bb8bbec0a1dc205a671. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aec442fecd64614858bce4a5bce5db4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:38.647526 8546 raft_consensus.cc:2932] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index -1 to 41, VOTER 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:38.647938 6908 raft_consensus.cc:2932] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, VOTER 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:38.650185 6632 catalog_manager.cc:5484] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b reported cstate change: config changed from index -1 to 41, VOTER 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:38.742512 6908 consensus_queue.cc:237] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:38.751425 6778 raft_consensus.cc:1261] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240503 13:59:38.751945 7659 consensus_queue.cc:1035] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 59, Time since last communication: 0.000s
I20240503 13:59:38.755930 7659 raft_consensus.cc:2932] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:38.757153 6775 raft_consensus.cc:2932] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:38.758505 6632 catalog_manager.cc:5484] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: config changed from index -1 to 61, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
W20240503 13:59:38.765802 6842 consensus_peers.cc:485] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 -> Peer 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579): Couldn't send request to peer 083b1c5c79a04af899eea33ba9f4502f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a9724b166dc4ca690fc912049779985. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:38.820183 7038 consensus_queue.cc:237] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:38.823416 6778 raft_consensus.cc:1261] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240503 13:59:38.823984 7819 consensus_queue.cc:1035] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20240503 13:59:38.826058 7819 raft_consensus.cc:2932] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:38.828088 6775 raft_consensus.cc:2932] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:38.828749 6635 catalog_manager.cc:5484] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b reported cstate change: config changed from index -1 to 39, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
W20240503 13:59:38.832047 6975 consensus_peers.cc:485] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b -> Peer 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135:34409): Couldn't send request to peer 4f82135907754dd8b69d08d6b7eae847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d118ea0cbd824762906095329740d0a3. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:38.854936 8738 raft_consensus.cc:482] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:38.856256 8738 raft_consensus.cc:504] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:38.857282 8738 leader_election.cc:288] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132:41965)
I20240503 13:59:38.868503 8023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d845691ee64bf3a5d6c2df59842a54" candidate_uuid: "083b1c5c79a04af899eea33ba9f4502f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" is_pre_election: true
I20240503 13:59:38.868769 8023 raft_consensus.cc:2454] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 083b1c5c79a04af899eea33ba9f4502f in term 1.
I20240503 13:59:38.869182 8221 leader_election.cc:302] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 083b1c5c79a04af899eea33ba9f4502f, 40e8eacd02b44bb8bbec0a1dc205a671; no voters:
I20240503 13:59:38.869443 8738 raft_consensus.cc:2781] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:59:38.869532 8738 raft_consensus.cc:482] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:38.869585 8738 raft_consensus.cc:3037] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:38.871949 8738 raft_consensus.cc:504] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:38.872242 8738 leader_election.cc:288] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [CANDIDATE]: Term 2 election: Requested vote from peers 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132:41965)
I20240503 13:59:38.872656 8023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d845691ee64bf3a5d6c2df59842a54" candidate_uuid: "083b1c5c79a04af899eea33ba9f4502f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "40e8eacd02b44bb8bbec0a1dc205a671"
I20240503 13:59:38.872762 8023 raft_consensus.cc:3037] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:38.875538 8023 raft_consensus.cc:2454] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 083b1c5c79a04af899eea33ba9f4502f in term 2.
I20240503 13:59:38.875823 8221 leader_election.cc:302] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 083b1c5c79a04af899eea33ba9f4502f, 40e8eacd02b44bb8bbec0a1dc205a671; no voters:
I20240503 13:59:38.877589 8738 raft_consensus.cc:2781] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:38.877760 8738 raft_consensus.cc:686] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 2 LEADER]: Becoming Leader. State: Replica: 083b1c5c79a04af899eea33ba9f4502f, State: Running, Role: LEADER
I20240503 13:59:38.880334 8738 consensus_queue.cc:237] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:38.883021 6632 catalog_manager.cc:5484] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "083b1c5c79a04af899eea33ba9f4502f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:38.906940 6776 consensus_queue.cc:237] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:38.912698 6906 raft_consensus.cc:1261] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Refusing update from remote peer 791cddb67b7e415aad7edb5151037418: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240503 13:59:38.913285 8496 consensus_queue.cc:1035] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20240503 13:59:38.915696 7663 raft_consensus.cc:2932] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index -1 to 63, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:38.916244 6906 raft_consensus.cc:2932] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index -1 to 63, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:38.922050 6632 catalog_manager.cc:5484] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: config changed from index -1 to 63, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "791cddb67b7e415aad7edb5151037418" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
W20240503 13:59:38.925055 6711 consensus_peers.cc:485] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 -> Peer 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133:39967): Couldn't send request to peer 4c5b7c28dd2a4920a2b4e97d1afe2384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527c7e7cf4f24ce3bed276c1860aa041. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:38.948493 8757 ts_tablet_manager.cc:902] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847: Initiating tablet copy from peer 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:38.949585 8757 tablet_copy_client.cc:308] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:40699
I20240503 13:59:38.961861 7060 tablet_copy_service.cc:140] P 14c195eab0b34945afd10f3291ed080b: Received BeginTabletCopySession request for tablet 9a52cb95109045d2bbfa3def26848224 from peer 4f82135907754dd8b69d08d6b7eae847 ({username='slave'} at 127.0.99.135:55545)
I20240503 13:59:38.962009 7060 tablet_copy_service.cc:161] P 14c195eab0b34945afd10f3291ed080b: Beginning new tablet copy session on tablet 9a52cb95109045d2bbfa3def26848224 from peer 4f82135907754dd8b69d08d6b7eae847 at {username='slave'} at 127.0.99.135:55545: session id = 4f82135907754dd8b69d08d6b7eae847-9a52cb95109045d2bbfa3def26848224
I20240503 13:59:38.963476 7060 tablet_copy_source_session.cc:205] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:38.967155 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a52cb95109045d2bbfa3def26848224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:38.976728 8757 tablet_copy_client.cc:783] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:38.976945 8757 tablet_copy_client.cc:647] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:38.978675 8757 tablet_copy_client.cc:523] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:38.981917 8757 tablet_bootstrap.cc:492] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap starting.
I20240503 13:59:39.002837 6906 consensus_queue.cc:237] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:39.006500 7038 raft_consensus.cc:1261] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240503 13:59:39.007095 7659 consensus_queue.cc:1035] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
W20240503 13:59:39.009407 6843 consensus_peers.cc:485] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 -> Peer 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132:41965): Couldn't send request to peer 40e8eacd02b44bb8bbec0a1dc205a671. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6409f411c5d24e97b34aa9cce7c5a5ea. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.009706 8470 raft_consensus.cc:2932] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index -1 to 70, VOTER 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:39.013314 6632 catalog_manager.cc:5484] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: config changed from index -1 to 70, VOTER 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:39.014870 8757 log.cc:824] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:39.015704 7039 raft_consensus.cc:2932] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index -1 to 70, VOTER 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:39.107877 8757 tablet_bootstrap.cc:492] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap replayed 1/1 log segments. Stats: ops{read=47 overwritten=0 applied=46 ignored=0} inserts{seen=677 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:59:39.108398 8757 tablet_bootstrap.cc:492] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap complete.
I20240503 13:59:39.108922 8757 ts_tablet_manager.cc:1367] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847: Time spent bootstrapping tablet: real 0.127s user 0.062s sys 0.004s
I20240503 13:59:39.111413 8757 raft_consensus.cc:348] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:39.112965 8757 raft_consensus.cc:729] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f82135907754dd8b69d08d6b7eae847, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.113490 8757 consensus_queue.cc:260] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 46, Last appended: 1.47, Last appended by leader: 47, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:39.114614 8757 ts_tablet_manager.cc:1398] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 13:59:39.115188 7060 tablet_copy_service.cc:342] P 14c195eab0b34945afd10f3291ed080b: Request end of tablet copy session 4f82135907754dd8b69d08d6b7eae847-9a52cb95109045d2bbfa3def26848224 received from {username='slave'} at 127.0.99.135:55545
I20240503 13:59:39.115324 7060 tablet_copy_service.cc:433] P 14c195eab0b34945afd10f3291ed080b: ending tablet copy session 4f82135907754dd8b69d08d6b7eae847-9a52cb95109045d2bbfa3def26848224 on tablet 9a52cb95109045d2bbfa3def26848224 with peer 4f82135907754dd8b69d08d6b7eae847
I20240503 13:59:39.132059 6778 consensus_queue.cc:237] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:39.134121 7039 raft_consensus.cc:1261] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Refusing update from remote peer 791cddb67b7e415aad7edb5151037418: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240503 13:59:39.137858 8496 consensus_queue.cc:1035] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 72, Time since last communication: 0.000s
W20240503 13:59:39.138566 6711 consensus_peers.cc:485] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 -> Peer 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579): Couldn't send request to peer 083b1c5c79a04af899eea33ba9f4502f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a871746624e442597588274959d5e24. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.143030 7038 raft_consensus.cc:1203] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.73] Dedup: 1.73->[]
I20240503 13:59:39.143113 8540 raft_consensus.cc:2932] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index -1 to 74, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:39.144558 7039 raft_consensus.cc:2932] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:39.147022 6632 catalog_manager.cc:5484] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b reported cstate change: config changed from index -1 to 74, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "791cddb67b7e415aad7edb5151037418" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:39.165130 8793 ts_tablet_manager.cc:902] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Initiating tablet copy from peer 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:39.166018 8793 tablet_copy_client.cc:308] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:40699
I20240503 13:59:39.176363 7060 tablet_copy_service.cc:140] P 14c195eab0b34945afd10f3291ed080b: Received BeginTabletCopySession request for tablet 57ea09e3f1f74d74839ddb0afd90acc2 from peer 4c5b7c28dd2a4920a2b4e97d1afe2384 ({username='slave'} at 127.0.99.133:60697)
I20240503 13:59:39.176486 7060 tablet_copy_service.cc:161] P 14c195eab0b34945afd10f3291ed080b: Beginning new tablet copy session on tablet 57ea09e3f1f74d74839ddb0afd90acc2 from peer 4c5b7c28dd2a4920a2b4e97d1afe2384 at {username='slave'} at 127.0.99.133:60697: session id = 4c5b7c28dd2a4920a2b4e97d1afe2384-57ea09e3f1f74d74839ddb0afd90acc2
I20240503 13:59:39.177428 7060 tablet_copy_source_session.cc:205] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:39.188741 8793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ea09e3f1f74d74839ddb0afd90acc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.212918 8757 ts_tablet_manager.cc:902] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847: Initiating tablet copy from peer 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:39.213351 8757 tablet_copy_client.cc:308] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:40699
I20240503 13:59:39.213971 8793 tablet_copy_client.cc:783] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:39.214205 8793 tablet_copy_client.cc:647] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:39.217803 8793 tablet_copy_client.cc:523] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:39.219781 8793 tablet_bootstrap.cc:492] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap starting.
I20240503 13:59:39.221203 7060 tablet_copy_service.cc:140] P 14c195eab0b34945afd10f3291ed080b: Received BeginTabletCopySession request for tablet d118ea0cbd824762906095329740d0a3 from peer 4f82135907754dd8b69d08d6b7eae847 ({username='slave'} at 127.0.99.135:55545)
I20240503 13:59:39.221295 7060 tablet_copy_service.cc:161] P 14c195eab0b34945afd10f3291ed080b: Beginning new tablet copy session on tablet d118ea0cbd824762906095329740d0a3 from peer 4f82135907754dd8b69d08d6b7eae847 at {username='slave'} at 127.0.99.135:55545: session id = 4f82135907754dd8b69d08d6b7eae847-d118ea0cbd824762906095329740d0a3
I20240503 13:59:39.222162 7060 tablet_copy_source_session.cc:205] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:39.223868 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d118ea0cbd824762906095329740d0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.222786 8813 ts_tablet_manager.cc:902] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671: Initiating tablet copy from peer 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:39.224819 8813 tablet_copy_client.cc:308] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:40699
I20240503 13:59:39.227622 8757 tablet_copy_client.cc:783] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:39.227813 8757 tablet_copy_client.cc:647] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:39.246711 8757 tablet_copy_client.cc:523] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:39.248222 8757 tablet_bootstrap.cc:492] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap starting.
I20240503 13:59:39.258692 8793 log.cc:824] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:39.276440 7060 tablet_copy_service.cc:140] P 14c195eab0b34945afd10f3291ed080b: Received BeginTabletCopySession request for tablet 4aec442fecd64614858bce4a5bce5db4 from peer 40e8eacd02b44bb8bbec0a1dc205a671 ({username='slave'} at 127.0.99.132:51615)
I20240503 13:59:39.276561 7060 tablet_copy_service.cc:161] P 14c195eab0b34945afd10f3291ed080b: Beginning new tablet copy session on tablet 4aec442fecd64614858bce4a5bce5db4 from peer 40e8eacd02b44bb8bbec0a1dc205a671 at {username='slave'} at 127.0.99.132:51615: session id = 40e8eacd02b44bb8bbec0a1dc205a671-4aec442fecd64614858bce4a5bce5db4
I20240503 13:59:39.277420 7060 tablet_copy_source_session.cc:205] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:39.278966 8813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aec442fecd64614858bce4a5bce5db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.282121 8813 tablet_copy_client.cc:783] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:39.282307 8813 tablet_copy_client.cc:647] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:39.284765 8813 tablet_copy_client.cc:523] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:39.286630 8813 tablet_bootstrap.cc:492] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap starting.
I20240503 13:59:39.304697 8023 raft_consensus.cc:1261] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 2 FOLLOWER]: Refusing update from remote peer 083b1c5c79a04af899eea33ba9f4502f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 13:59:39.305382 8745 consensus_queue.cc:1035] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:39.314543 6777 consensus_queue.cc:237] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:39.320012 7039 raft_consensus.cc:1261] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Refusing update from remote peer 791cddb67b7e415aad7edb5151037418: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240503 13:59:39.323212 8540 consensus_queue.cc:1035] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20240503 13:59:39.329242 8496 raft_consensus.cc:2932] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index -1 to 86, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
W20240503 13:59:39.330721 6713 consensus_peers.cc:485] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 -> Peer 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135:34409): Couldn't send request to peer 4f82135907754dd8b69d08d6b7eae847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fb5bbc56575455eb122d1e401fbdc64. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.333293 7039 raft_consensus.cc:2932] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index -1 to 86, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:39.333701 6635 catalog_manager.cc:5484] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 reported cstate change: config changed from index -1 to 86, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "791cddb67b7e415aad7edb5151037418" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:39.360965 8757 tablet_bootstrap.cc:492] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap replayed 1/1 log segments. Stats: ops{read=51 overwritten=0 applied=51 ignored=0} inserts{seen=682 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:39.365650 8757 tablet_bootstrap.cc:492] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap complete.
I20240503 13:59:39.365916 8757 ts_tablet_manager.cc:1367] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847: Time spent bootstrapping tablet: real 0.118s user 0.055s sys 0.000s
I20240503 13:59:39.366591 8757 raft_consensus.cc:348] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:39.367867 8757 raft_consensus.cc:729] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f82135907754dd8b69d08d6b7eae847, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.368077 8757 consensus_queue.cc:260] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 1.51, Last appended by leader: 51, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:39.369885 8757 ts_tablet_manager.cc:1398] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:39.371506 7060 tablet_copy_service.cc:342] P 14c195eab0b34945afd10f3291ed080b: Request end of tablet copy session 4f82135907754dd8b69d08d6b7eae847-d118ea0cbd824762906095329740d0a3 received from {username='slave'} at 127.0.99.135:55545
I20240503 13:59:39.371587 7060 tablet_copy_service.cc:433] P 14c195eab0b34945afd10f3291ed080b: ending tablet copy session 4f82135907754dd8b69d08d6b7eae847-d118ea0cbd824762906095329740d0a3 on tablet d118ea0cbd824762906095329740d0a3 with peer 4f82135907754dd8b69d08d6b7eae847
I20240503 13:59:39.409168 8839 ts_tablet_manager.cc:902] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f: Initiating tablet copy from peer 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:39.410051 8839 tablet_copy_client.cc:308] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33439
I20240503 13:59:39.423638 8793 tablet_bootstrap.cc:492] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap replayed 1/1 log segments. Stats: ops{read=83 overwritten=0 applied=83 ignored=0} inserts{seen=714 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:39.424211 8793 tablet_bootstrap.cc:492] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap complete.
I20240503 13:59:39.424631 8793 ts_tablet_manager.cc:1367] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Time spent bootstrapping tablet: real 0.205s user 0.077s sys 0.011s
I20240503 13:59:39.426492 6907 consensus_queue.cc:237] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:39.427959 7038 raft_consensus.cc:1261] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 13:59:39.428543 7659 consensus_queue.cc:1035] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 13:59:39.429276 8793 raft_consensus.cc:348] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:39.429533 8793 raft_consensus.cc:729] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5b7c28dd2a4920a2b4e97d1afe2384, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.431222 7659 raft_consensus.cc:2932] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index -1 to 82, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:39.431499 7039 raft_consensus.cc:2932] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index -1 to 82, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:39.431722 8793 consensus_queue.cc:260] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 83, Last appended: 1.83, Last appended by leader: 83, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:39.432724 8793 ts_tablet_manager.cc:1398] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240503 13:59:39.433881 6632 catalog_manager.cc:5484] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: config changed from index -1 to 82, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:39.440209 8421 raft_consensus.cc:1203] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.26->[1.27-1.63] Dedup: 1.47->[1.48-1.63]
I20240503 13:59:39.441175 7060 tablet_copy_service.cc:342] P 14c195eab0b34945afd10f3291ed080b: Request end of tablet copy session 4c5b7c28dd2a4920a2b4e97d1afe2384-57ea09e3f1f74d74839ddb0afd90acc2 received from {username='slave'} at 127.0.99.133:60697
I20240503 13:59:39.441241 7060 tablet_copy_service.cc:433] P 14c195eab0b34945afd10f3291ed080b: ending tablet copy session 4c5b7c28dd2a4920a2b4e97d1afe2384-57ea09e3f1f74d74839ddb0afd90acc2 on tablet 57ea09e3f1f74d74839ddb0afd90acc2 with peer 4c5b7c28dd2a4920a2b4e97d1afe2384
I20240503 13:59:39.445080 8793 ts_tablet_manager.cc:902] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Initiating tablet copy from peer 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:39.445533 8793 tablet_copy_client.cc:308] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43893
W20240503 13:59:39.447353 6842 consensus_peers.cc:485] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 -> Peer 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133:39967): Couldn't send request to peer 4c5b7c28dd2a4920a2b4e97d1afe2384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2b880960ed44b20b81ddf91c8534638. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.452478 6798 tablet_copy_service.cc:140] P 791cddb67b7e415aad7edb5151037418: Received BeginTabletCopySession request for tablet 527c7e7cf4f24ce3bed276c1860aa041 from peer 4c5b7c28dd2a4920a2b4e97d1afe2384 ({username='slave'} at 127.0.99.133:34371)
I20240503 13:59:39.452584 6929 tablet_copy_service.cc:140] P 52a58bb8fe574f5ab1485c59bf994570: Received BeginTabletCopySession request for tablet 4a9724b166dc4ca690fc912049779985 from peer 083b1c5c79a04af899eea33ba9f4502f ({username='slave'} at 127.0.99.134:42783)
I20240503 13:59:39.452625 6798 tablet_copy_service.cc:161] P 791cddb67b7e415aad7edb5151037418: Beginning new tablet copy session on tablet 527c7e7cf4f24ce3bed276c1860aa041 from peer 4c5b7c28dd2a4920a2b4e97d1afe2384 at {username='slave'} at 127.0.99.133:34371: session id = 4c5b7c28dd2a4920a2b4e97d1afe2384-527c7e7cf4f24ce3bed276c1860aa041
I20240503 13:59:39.452679 6929 tablet_copy_service.cc:161] P 52a58bb8fe574f5ab1485c59bf994570: Beginning new tablet copy session on tablet 4a9724b166dc4ca690fc912049779985 from peer 083b1c5c79a04af899eea33ba9f4502f at {username='slave'} at 127.0.99.134:42783: session id = 083b1c5c79a04af899eea33ba9f4502f-4a9724b166dc4ca690fc912049779985
I20240503 13:59:39.454022 6929 tablet_copy_source_session.cc:205] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:39.454017 6798 tablet_copy_source_session.cc:205] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:39.456574 8839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9724b166dc4ca690fc912049779985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.459784 8793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527c7e7cf4f24ce3bed276c1860aa041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.461948 8793 tablet_copy_client.cc:783] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:39.462128 8793 tablet_copy_client.cc:647] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:39.463362 8839 tablet_copy_client.cc:783] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:39.463609 8839 tablet_copy_client.cc:647] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:39.464011 8813 tablet_bootstrap.cc:492] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap replayed 1/1 log segments. Stats: ops{read=64 overwritten=0 applied=63 ignored=0} inserts{seen=735 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:59:39.464442 8813 tablet_bootstrap.cc:492] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap complete.
I20240503 13:59:39.464702 8813 ts_tablet_manager.cc:1367] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671: Time spent bootstrapping tablet: real 0.178s user 0.062s sys 0.007s
I20240503 13:59:39.465286 8813 raft_consensus.cc:348] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:39.466761 8839 tablet_copy_client.cc:523] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:39.466821 8813 raft_consensus.cc:729] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e8eacd02b44bb8bbec0a1dc205a671, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.466989 8813 consensus_queue.cc:260] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 1.64, Last appended by leader: 64, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:39.467222 8813 ts_tablet_manager.cc:1398] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:39.468261 8839 tablet_bootstrap.cc:492] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap starting.
I20240503 13:59:39.470822 7060 tablet_copy_service.cc:342] P 14c195eab0b34945afd10f3291ed080b: Request end of tablet copy session 40e8eacd02b44bb8bbec0a1dc205a671-4aec442fecd64614858bce4a5bce5db4 received from {username='slave'} at 127.0.99.132:51615
I20240503 13:59:39.470898 7060 tablet_copy_service.cc:433] P 14c195eab0b34945afd10f3291ed080b: ending tablet copy session 40e8eacd02b44bb8bbec0a1dc205a671-4aec442fecd64614858bce4a5bce5db4 on tablet 4aec442fecd64614858bce4a5bce5db4 with peer 40e8eacd02b44bb8bbec0a1dc205a671
I20240503 13:59:39.471275 8793 tablet_copy_client.cc:523] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:39.472585 8793 tablet_bootstrap.cc:492] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap starting.
I20240503 13:59:39.520646 7036 consensus_queue.cc:237] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
W20240503 13:59:39.522742 6974 consensus_peers.cc:485] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b -> Peer 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132:41965): Couldn't send request to peer 40e8eacd02b44bb8bbec0a1dc205a671. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e026fe8a768941ce84378f23b3dabff2. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.526962 6907 raft_consensus.cc:1261] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240503 13:59:39.528331 7819 consensus_queue.cc:1035] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 13:59:39.530354 8680 raft_consensus.cc:2932] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index -1 to 94, VOTER 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:39.531232 6906 raft_consensus.cc:2932] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index -1 to 94, VOTER 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:39.533972 8813 ts_tablet_manager.cc:902] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671: Initiating tablet copy from peer 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:39.534201 6635 catalog_manager.cc:5484] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b reported cstate change: config changed from index -1 to 94, VOTER 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:39.534456 8813 tablet_copy_client.cc:308] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33439
I20240503 13:59:39.538960 6929 tablet_copy_service.cc:140] P 52a58bb8fe574f5ab1485c59bf994570: Received BeginTabletCopySession request for tablet 6409f411c5d24e97b34aa9cce7c5a5ea from peer 40e8eacd02b44bb8bbec0a1dc205a671 ({username='slave'} at 127.0.99.132:39753)
I20240503 13:59:39.539047 6929 tablet_copy_service.cc:161] P 52a58bb8fe574f5ab1485c59bf994570: Beginning new tablet copy session on tablet 6409f411c5d24e97b34aa9cce7c5a5ea from peer 40e8eacd02b44bb8bbec0a1dc205a671 at {username='slave'} at 127.0.99.132:39753: session id = 40e8eacd02b44bb8bbec0a1dc205a671-6409f411c5d24e97b34aa9cce7c5a5ea
I20240503 13:59:39.539970 6929 tablet_copy_source_session.cc:205] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:39.540956 8813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6409f411c5d24e97b34aa9cce7c5a5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.542837 8813 tablet_copy_client.cc:783] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:39.542959 8813 tablet_copy_client.cc:647] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:39.576782 8813 tablet_copy_client.cc:523] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:39.578344 8813 tablet_bootstrap.cc:492] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap starting.
I20240503 13:59:39.613229 8793 tablet_bootstrap.cc:492] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap replayed 1/1 log segments. Stats: ops{read=73 overwritten=0 applied=72 ignored=0} inserts{seen=703 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:59:39.613622 8793 tablet_bootstrap.cc:492] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap complete.
I20240503 13:59:39.613888 8793 ts_tablet_manager.cc:1367] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Time spent bootstrapping tablet: real 0.141s user 0.073s sys 0.000s
I20240503 13:59:39.614598 8793 raft_consensus.cc:348] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:39.616101 8793 raft_consensus.cc:729] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5b7c28dd2a4920a2b4e97d1afe2384, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.616333 8793 consensus_queue.cc:260] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 72, Last appended: 1.73, Last appended by leader: 73, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:39.616690 8793 ts_tablet_manager.cc:1398] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:39.621773 6798 tablet_copy_service.cc:342] P 791cddb67b7e415aad7edb5151037418: Request end of tablet copy session 4c5b7c28dd2a4920a2b4e97d1afe2384-527c7e7cf4f24ce3bed276c1860aa041 received from {username='slave'} at 127.0.99.133:34371
I20240503 13:59:39.621867 6798 tablet_copy_service.cc:433] P 791cddb67b7e415aad7edb5151037418: ending tablet copy session 4c5b7c28dd2a4920a2b4e97d1afe2384-527c7e7cf4f24ce3bed276c1860aa041 on tablet 527c7e7cf4f24ce3bed276c1860aa041 with peer 4c5b7c28dd2a4920a2b4e97d1afe2384
I20240503 13:59:39.637614 8421 raft_consensus.cc:1203] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.38->[1.39-1.68] Dedup: 1.51->[1.52-1.68]
I20240503 13:59:39.656006 8023 raft_consensus.cc:1203] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.74] Dedup: 1.64->[1.65-1.74]
I20240503 13:59:39.698824 6907 consensus_queue.cc:237] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
W20240503 13:59:39.702347 6842 consensus_peers.cc:485] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 -> Peer 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579): Couldn't send request to peer 083b1c5c79a04af899eea33ba9f4502f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70583380c4e24b4694fc5477d880d944. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.702760 6778 raft_consensus.cc:1261] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240503 13:59:39.705430 8554 consensus_queue.cc:1035] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 78, Time since last communication: 0.000s
I20240503 13:59:39.711477 8548 raft_consensus.cc:2932] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index -1 to 80, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:39.713609 6776 raft_consensus.cc:2932] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index -1 to 80, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:39.713905 6635 catalog_manager.cc:5484] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: config changed from index -1 to 80, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:39.718756 8156 raft_consensus.cc:1203] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.106] Dedup: 1.83->[1.84-1.106]
I20240503 13:59:39.746587 8923 ts_tablet_manager.cc:902] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f: Initiating tablet copy from peer 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:39.747256 8923 tablet_copy_client.cc:308] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43893
I20240503 13:59:39.748396 8757 ts_tablet_manager.cc:902] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847: Initiating tablet copy from peer 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:39.748843 8757 tablet_copy_client.cc:308] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43893
I20240503 13:59:39.751240 8813 tablet_bootstrap.cc:492] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=79 ignored=0} inserts{seen=751 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:59:39.751559 8813 tablet_bootstrap.cc:492] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap complete.
I20240503 13:59:39.751744 8813 ts_tablet_manager.cc:1367] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671: Time spent bootstrapping tablet: real 0.173s user 0.064s sys 0.012s
I20240503 13:59:39.752166 8813 raft_consensus.cc:348] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:39.752629 8813 raft_consensus.cc:729] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e8eacd02b44bb8bbec0a1dc205a671, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.752776 8813 consensus_queue.cc:260] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 79, Last appended: 1.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:39.757323 8813 ts_tablet_manager.cc:1398] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 13:59:39.759016 6929 tablet_copy_service.cc:342] P 52a58bb8fe574f5ab1485c59bf994570: Request end of tablet copy session 40e8eacd02b44bb8bbec0a1dc205a671-6409f411c5d24e97b34aa9cce7c5a5ea received from {username='slave'} at 127.0.99.132:39753
I20240503 13:59:39.759109 6929 tablet_copy_service.cc:433] P 52a58bb8fe574f5ab1485c59bf994570: ending tablet copy session 40e8eacd02b44bb8bbec0a1dc205a671-6409f411c5d24e97b34aa9cce7c5a5ea on tablet 6409f411c5d24e97b34aa9cce7c5a5ea with peer 40e8eacd02b44bb8bbec0a1dc205a671
I20240503 13:59:39.763378 6798 tablet_copy_service.cc:140] P 791cddb67b7e415aad7edb5151037418: Received BeginTabletCopySession request for tablet 5fb5bbc56575455eb122d1e401fbdc64 from peer 4f82135907754dd8b69d08d6b7eae847 ({username='slave'} at 127.0.99.135:33591)
I20240503 13:59:39.763463 6798 tablet_copy_service.cc:161] P 791cddb67b7e415aad7edb5151037418: Beginning new tablet copy session on tablet 5fb5bbc56575455eb122d1e401fbdc64 from peer 4f82135907754dd8b69d08d6b7eae847 at {username='slave'} at 127.0.99.135:33591: session id = 4f82135907754dd8b69d08d6b7eae847-5fb5bbc56575455eb122d1e401fbdc64
I20240503 13:59:39.764411 6798 tablet_copy_source_session.cc:205] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:39.765204 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb5bbc56575455eb122d1e401fbdc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.767765 6798 tablet_copy_service.cc:140] P 791cddb67b7e415aad7edb5151037418: Received BeginTabletCopySession request for tablet 0a871746624e442597588274959d5e24 from peer 083b1c5c79a04af899eea33ba9f4502f ({username='slave'} at 127.0.99.134:36761)
I20240503 13:59:39.767848 6798 tablet_copy_service.cc:161] P 791cddb67b7e415aad7edb5151037418: Beginning new tablet copy session on tablet 0a871746624e442597588274959d5e24 from peer 083b1c5c79a04af899eea33ba9f4502f at {username='slave'} at 127.0.99.134:36761: session id = 083b1c5c79a04af899eea33ba9f4502f-0a871746624e442597588274959d5e24
I20240503 13:59:39.768543 6798 tablet_copy_source_session.cc:205] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:39.769735 8757 tablet_copy_client.cc:783] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:39.769888 8757 tablet_copy_client.cc:647] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:39.771190 8757 tablet_copy_client.cc:523] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:39.772451 8757 tablet_bootstrap.cc:492] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap starting.
I20240503 13:59:39.776348 8923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a871746624e442597588274959d5e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.780112 8923 tablet_copy_client.cc:783] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:39.780315 8923 tablet_copy_client.cc:647] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:39.811266 6907 consensus_queue.cc:237] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.91, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:39.815486 6777 raft_consensus.cc:1261] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
W20240503 13:59:39.816684 6844 consensus_peers.cc:485] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 -> Peer 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135:34409): Couldn't send request to peer 4f82135907754dd8b69d08d6b7eae847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f882dbedd134f2998642ac747567db0. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.818311 8548 consensus_queue.cc:1035] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 90, Time since last communication: 0.000s
I20240503 13:59:39.818810 8839 tablet_bootstrap.cc:492] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap replayed 1/1 log segments. Stats: ops{read=87 overwritten=0 applied=87 ignored=0} inserts{seen=772 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:39.819128 8839 tablet_bootstrap.cc:492] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap complete.
I20240503 13:59:39.819326 8839 ts_tablet_manager.cc:1367] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f: Time spent bootstrapping tablet: real 0.351s user 0.085s sys 0.003s
I20240503 13:59:39.819903 8839 raft_consensus.cc:348] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:39.820008 8839 raft_consensus.cc:729] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083b1c5c79a04af899eea33ba9f4502f, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.820200 8839 consensus_queue.cc:260] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 87, Last appended: 1.87, Last appended by leader: 87, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:39.820995 8839 ts_tablet_manager.cc:1398] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:39.822636 6929 tablet_copy_service.cc:342] P 52a58bb8fe574f5ab1485c59bf994570: Request end of tablet copy session 083b1c5c79a04af899eea33ba9f4502f-4a9724b166dc4ca690fc912049779985 received from {username='slave'} at 127.0.99.134:42783
I20240503 13:59:39.822705 6929 tablet_copy_service.cc:433] P 52a58bb8fe574f5ab1485c59bf994570: ending tablet copy session 083b1c5c79a04af899eea33ba9f4502f-4a9724b166dc4ca690fc912049779985 on tablet 4a9724b166dc4ca690fc912049779985 with peer 083b1c5c79a04af899eea33ba9f4502f
I20240503 13:59:39.829492 8917 raft_consensus.cc:2932] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.92: config changed from index -1 to 92, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:39.832772 6776 raft_consensus.cc:2932] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index -1 to 92, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:39.836848 6635 catalog_manager.cc:5484] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 reported cstate change: config changed from index -1 to 92, VOTER 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:39.851485 8923 tablet_copy_client.cc:523] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:39.867455 8793 ts_tablet_manager.cc:902] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Initiating tablet copy from peer 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:39.868000 8793 tablet_copy_client.cc:308] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33439
I20240503 13:59:39.871006 8288 raft_consensus.cc:1203] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.101] Dedup: 1.87->[1.88-1.101]
I20240503 13:59:39.884477 6929 tablet_copy_service.cc:140] P 52a58bb8fe574f5ab1485c59bf994570: Received BeginTabletCopySession request for tablet f2b880960ed44b20b81ddf91c8534638 from peer 4c5b7c28dd2a4920a2b4e97d1afe2384 ({username='slave'} at 127.0.99.133:49975)
I20240503 13:59:39.884581 6929 tablet_copy_service.cc:161] P 52a58bb8fe574f5ab1485c59bf994570: Beginning new tablet copy session on tablet f2b880960ed44b20b81ddf91c8534638 from peer 4c5b7c28dd2a4920a2b4e97d1afe2384 at {username='slave'} at 127.0.99.133:49975: session id = 4c5b7c28dd2a4920a2b4e97d1afe2384-f2b880960ed44b20b81ddf91c8534638
I20240503 13:59:39.885499 6929 tablet_copy_source_session.cc:205] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:39.887022 8793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b880960ed44b20b81ddf91c8534638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:39.890084 8793 tablet_copy_client.cc:783] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:39.890236 8793 tablet_copy_client.cc:647] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:39.893040 8793 tablet_copy_client.cc:523] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:39.894160 8793 tablet_bootstrap.cc:492] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap starting.
I20240503 13:59:39.906854 8923 tablet_bootstrap.cc:492] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap starting.
I20240503 13:59:39.921329 8156 raft_consensus.cc:1203] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.87] Dedup: 1.73->[1.74-1.87]
I20240503 13:59:39.947340 8757 tablet_bootstrap.cc:492] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=0 applied=97 ignored=0} inserts{seen=728 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:39.947666 8757 tablet_bootstrap.cc:492] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap complete.
I20240503 13:59:39.947852 8757 ts_tablet_manager.cc:1367] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847: Time spent bootstrapping tablet: real 0.175s user 0.075s sys 0.012s
I20240503 13:59:39.948282 8757 raft_consensus.cc:348] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:39.948410 8757 raft_consensus.cc:729] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f82135907754dd8b69d08d6b7eae847, State: Initialized, Role: FOLLOWER
I20240503 13:59:39.948551 8757 consensus_queue.cc:260] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 1.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:39.950280 8757 ts_tablet_manager.cc:1398] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:39.952092 6798 tablet_copy_service.cc:342] P 791cddb67b7e415aad7edb5151037418: Request end of tablet copy session 4f82135907754dd8b69d08d6b7eae847-5fb5bbc56575455eb122d1e401fbdc64 received from {username='slave'} at 127.0.99.135:33591
I20240503 13:59:39.952163 6798 tablet_copy_service.cc:433] P 791cddb67b7e415aad7edb5151037418: ending tablet copy session 4f82135907754dd8b69d08d6b7eae847-5fb5bbc56575455eb122d1e401fbdc64 on tablet 5fb5bbc56575455eb122d1e401fbdc64 with peer 4f82135907754dd8b69d08d6b7eae847
I20240503 13:59:39.957363 6776 consensus_queue.cc:237] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
W20240503 13:59:39.958698 6711 consensus_peers.cc:485] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 -> Peer 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133:39967): Couldn't send request to peer 4c5b7c28dd2a4920a2b4e97d1afe2384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a82b94a4baa84d778cc280d58c18af5b. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:39.959319 6906 raft_consensus.cc:1261] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Refusing update from remote peer 791cddb67b7e415aad7edb5151037418: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240503 13:59:39.961030 7663 consensus_queue.cc:1035] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240503 13:59:39.962383 8606 raft_consensus.cc:2932] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index -1 to 93, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:39.962731 6906 raft_consensus.cc:2932] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index -1 to 93, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:39.966221 6636 catalog_manager.cc:5484] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: config changed from index -1 to 93, VOTER 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "791cddb67b7e415aad7edb5151037418" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:40.008641 8793 tablet_bootstrap.cc:492] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap replayed 1/1 log segments. Stats: ops{read=98 overwritten=0 applied=98 ignored=0} inserts{seen=770 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.008986 8793 tablet_bootstrap.cc:492] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap complete.
I20240503 13:59:40.009202 8793 ts_tablet_manager.cc:1367] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Time spent bootstrapping tablet: real 0.115s user 0.081s sys 0.008s
I20240503 13:59:40.009634 8793 raft_consensus.cc:348] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:40.009747 8793 raft_consensus.cc:729] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5b7c28dd2a4920a2b4e97d1afe2384, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.009896 8793 consensus_queue.cc:260] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 98, Last appended: 1.98, Last appended by leader: 98, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:40.011803 8793 ts_tablet_manager.cc:1398] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:40.017132 6929 tablet_copy_service.cc:342] P 52a58bb8fe574f5ab1485c59bf994570: Request end of tablet copy session 4c5b7c28dd2a4920a2b4e97d1afe2384-f2b880960ed44b20b81ddf91c8534638 received from {username='slave'} at 127.0.99.133:49975
I20240503 13:59:40.017213 6929 tablet_copy_service.cc:433] P 52a58bb8fe574f5ab1485c59bf994570: ending tablet copy session 4c5b7c28dd2a4920a2b4e97d1afe2384-f2b880960ed44b20b81ddf91c8534638 on tablet f2b880960ed44b20b81ddf91c8534638 with peer 4c5b7c28dd2a4920a2b4e97d1afe2384
I20240503 13:59:40.049413 6906 consensus_queue.cc:237] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
W20240503 13:59:40.051139 6842 consensus_peers.cc:485] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 -> Peer 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579): Couldn't send request to peer 083b1c5c79a04af899eea33ba9f4502f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef2d39830da24221a3dc44da57c0a6e8. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:40.053671 7038 raft_consensus.cc:1261] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20240503 13:59:40.054677 8553 consensus_queue.cc:1035] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20240503 13:59:40.058537 8553 raft_consensus.cc:2932] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index -1 to 89, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:40.059521 7039 raft_consensus.cc:2932] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index -1 to 89, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:40.061688 6635 catalog_manager.cc:5484] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: config changed from index -1 to 89, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:40.067210 8923 tablet_bootstrap.cc:492] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=89 ignored=0} inserts{seen=774 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:59:40.067464 8023 raft_consensus.cc:1203] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.92] Dedup: 1.80->[1.81-1.92]
I20240503 13:59:40.067643 8923 tablet_bootstrap.cc:492] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap complete.
I20240503 13:59:40.067862 8923 ts_tablet_manager.cc:1367] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f: Time spent bootstrapping tablet: real 0.161s user 0.079s sys 0.006s
I20240503 13:59:40.068356 8923 raft_consensus.cc:348] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:40.068964 8923 raft_consensus.cc:729] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083b1c5c79a04af899eea33ba9f4502f, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.069180 8923 consensus_queue.cc:260] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 89, Last appended: 1.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:40.070892 8923 ts_tablet_manager.cc:1398] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 13:59:40.073487 6798 tablet_copy_service.cc:342] P 791cddb67b7e415aad7edb5151037418: Request end of tablet copy session 083b1c5c79a04af899eea33ba9f4502f-0a871746624e442597588274959d5e24 received from {username='slave'} at 127.0.99.134:36761
I20240503 13:59:40.073571 6798 tablet_copy_service.cc:433] P 791cddb67b7e415aad7edb5151037418: ending tablet copy session 083b1c5c79a04af899eea33ba9f4502f-0a871746624e442597588274959d5e24 on tablet 0a871746624e442597588274959d5e24 with peer 083b1c5c79a04af899eea33ba9f4502f
I20240503 13:59:40.132562 8813 ts_tablet_manager.cc:902] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671: Initiating tablet copy from peer 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:40.132982 8813 tablet_copy_client.cc:308] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:40699
I20240503 13:59:40.133420 7060 tablet_copy_service.cc:140] P 14c195eab0b34945afd10f3291ed080b: Received BeginTabletCopySession request for tablet e026fe8a768941ce84378f23b3dabff2 from peer 40e8eacd02b44bb8bbec0a1dc205a671 ({username='slave'} at 127.0.99.132:51615)
I20240503 13:59:40.133517 7060 tablet_copy_service.cc:161] P 14c195eab0b34945afd10f3291ed080b: Beginning new tablet copy session on tablet e026fe8a768941ce84378f23b3dabff2 from peer 40e8eacd02b44bb8bbec0a1dc205a671 at {username='slave'} at 127.0.99.132:51615: session id = 40e8eacd02b44bb8bbec0a1dc205a671-e026fe8a768941ce84378f23b3dabff2
I20240503 13:59:40.134364 7060 tablet_copy_source_session.cc:205] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.135197 8813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e026fe8a768941ce84378f23b3dabff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.137964 8813 tablet_copy_client.cc:783] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.138118 8813 tablet_copy_client.cc:647] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.144924 8813 tablet_copy_client.cc:523] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.146189 8813 tablet_bootstrap.cc:492] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap starting.
I20240503 13:59:40.153512 8923 ts_tablet_manager.cc:902] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f: Initiating tablet copy from peer 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:40.153931 8923 tablet_copy_client.cc:308] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33439
I20240503 13:59:40.161000 6929 tablet_copy_service.cc:140] P 52a58bb8fe574f5ab1485c59bf994570: Received BeginTabletCopySession request for tablet 70583380c4e24b4694fc5477d880d944 from peer 083b1c5c79a04af899eea33ba9f4502f ({username='slave'} at 127.0.99.134:42783)
I20240503 13:59:40.161089 6929 tablet_copy_service.cc:161] P 52a58bb8fe574f5ab1485c59bf994570: Beginning new tablet copy session on tablet 70583380c4e24b4694fc5477d880d944 from peer 083b1c5c79a04af899eea33ba9f4502f at {username='slave'} at 127.0.99.134:42783: session id = 083b1c5c79a04af899eea33ba9f4502f-70583380c4e24b4694fc5477d880d944
I20240503 13:59:40.161917 6929 tablet_copy_source_session.cc:205] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.164170 8923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70583380c4e24b4694fc5477d880d944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.166275 8923 tablet_copy_client.cc:783] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.166415 8923 tablet_copy_client.cc:647] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.191486 8923 tablet_copy_client.cc:523] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.192571 8923 tablet_bootstrap.cc:492] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap starting.
I20240503 13:59:40.194545 8757 ts_tablet_manager.cc:902] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847: Initiating tablet copy from peer 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:40.194983 8757 tablet_copy_client.cc:308] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33439
I20240503 13:59:40.199534 6929 tablet_copy_service.cc:140] P 52a58bb8fe574f5ab1485c59bf994570: Received BeginTabletCopySession request for tablet 4f882dbedd134f2998642ac747567db0 from peer 4f82135907754dd8b69d08d6b7eae847 ({username='slave'} at 127.0.99.135:45715)
I20240503 13:59:40.199625 6929 tablet_copy_service.cc:161] P 52a58bb8fe574f5ab1485c59bf994570: Beginning new tablet copy session on tablet 4f882dbedd134f2998642ac747567db0 from peer 4f82135907754dd8b69d08d6b7eae847 at {username='slave'} at 127.0.99.135:45715: session id = 4f82135907754dd8b69d08d6b7eae847-4f882dbedd134f2998642ac747567db0
I20240503 13:59:40.200390 6929 tablet_copy_source_session.cc:205] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.201082 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f882dbedd134f2998642ac747567db0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.203007 8757 tablet_copy_client.cc:783] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.203161 8757 tablet_copy_client.cc:647] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.208879 8757 tablet_copy_client.cc:523] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.209837 8757 tablet_bootstrap.cc:492] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap starting.
I20240503 13:59:40.217998 8420 raft_consensus.cc:1203] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.85->[1.86-1.106] Dedup: 1.97->[1.98-1.106]
I20240503 13:59:40.283703 8288 raft_consensus.cc:1203] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.103] Dedup: 1.90->[1.91-1.103]
I20240503 13:59:40.341907 8813 tablet_bootstrap.cc:492] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap replayed 1/1 log segments. Stats: ops{read=108 overwritten=0 applied=107 ignored=0} inserts{seen=779 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:59:40.343622 8813 tablet_bootstrap.cc:492] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671: Bootstrap complete.
I20240503 13:59:40.344877 8813 ts_tablet_manager.cc:1367] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671: Time spent bootstrapping tablet: real 0.199s user 0.083s sys 0.012s
I20240503 13:59:40.345350 8813 raft_consensus.cc:348] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:40.345614 8813 raft_consensus.cc:729] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e8eacd02b44bb8bbec0a1dc205a671, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.345742 8813 consensus_queue.cc:260] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 107, Last appended: 1.108, Last appended by leader: 108, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:40.345978 8813 ts_tablet_manager.cc:1398] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:40.346451 7060 tablet_copy_service.cc:342] P 14c195eab0b34945afd10f3291ed080b: Request end of tablet copy session 40e8eacd02b44bb8bbec0a1dc205a671-e026fe8a768941ce84378f23b3dabff2 received from {username='slave'} at 127.0.99.132:51615
I20240503 13:59:40.346521 7060 tablet_copy_service.cc:433] P 14c195eab0b34945afd10f3291ed080b: ending tablet copy session 40e8eacd02b44bb8bbec0a1dc205a671-e026fe8a768941ce84378f23b3dabff2 on tablet e026fe8a768941ce84378f23b3dabff2 with peer 40e8eacd02b44bb8bbec0a1dc205a671
I20240503 13:59:40.389389 8156 raft_consensus.cc:1203] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.81->[1.82-1.114] Dedup: 1.98->[1.99-1.114]
I20240503 13:59:40.420105 8793 ts_tablet_manager.cc:902] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384: Initiating tablet copy from peer 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:40.420548 8793 tablet_copy_client.cc:308] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:43893
I20240503 13:59:40.424631 6798 tablet_copy_service.cc:140] P 791cddb67b7e415aad7edb5151037418: Received BeginTabletCopySession request for tablet a82b94a4baa84d778cc280d58c18af5b from peer 4c5b7c28dd2a4920a2b4e97d1afe2384 ({username='slave'} at 127.0.99.133:34371)
I20240503 13:59:40.424734 6798 tablet_copy_service.cc:161] P 791cddb67b7e415aad7edb5151037418: Beginning new tablet copy session on tablet a82b94a4baa84d778cc280d58c18af5b from peer 4c5b7c28dd2a4920a2b4e97d1afe2384 at {username='slave'} at 127.0.99.133:34371: session id = 4c5b7c28dd2a4920a2b4e97d1afe2384-a82b94a4baa84d778cc280d58c18af5b
I20240503 13:59:40.425654 6798 tablet_copy_source_session.cc:205] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.426898 8793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82b94a4baa84d778cc280d58c18af5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.428818 8793 tablet_copy_client.cc:783] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.428970 8793 tablet_copy_client.cc:647] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.437253 8793 tablet_copy_client.cc:523] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.438432 8793 tablet_bootstrap.cc:492] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap starting.
I20240503 13:59:40.493911 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:40.511260 6889 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:40.515144 6758 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:40.534970 8923 tablet_bootstrap.cc:492] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=91 ignored=0} inserts{seen=776 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.535282 8923 tablet_bootstrap.cc:492] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap complete.
I20240503 13:59:40.535503 8923 ts_tablet_manager.cc:1367] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f: Time spent bootstrapping tablet: real 0.343s user 0.080s sys 0.001s
I20240503 13:59:40.535866 8923 raft_consensus.cc:348] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:40.535962 8923 raft_consensus.cc:729] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083b1c5c79a04af899eea33ba9f4502f, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.536087 8923 consensus_queue.cc:260] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 1.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:40.536688 8923 ts_tablet_manager.cc:1398] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.544329 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:40.548105 6929 tablet_copy_service.cc:342] P 52a58bb8fe574f5ab1485c59bf994570: Request end of tablet copy session 083b1c5c79a04af899eea33ba9f4502f-70583380c4e24b4694fc5477d880d944 received from {username='slave'} at 127.0.99.134:42783
I20240503 13:59:40.548182 6929 tablet_copy_service.cc:433] P 52a58bb8fe574f5ab1485c59bf994570: ending tablet copy session 083b1c5c79a04af899eea33ba9f4502f-70583380c4e24b4694fc5477d880d944 on tablet 70583380c4e24b4694fc5477d880d944 with peer 083b1c5c79a04af899eea33ba9f4502f
I20240503 13:59:40.557009 8923 ts_tablet_manager.cc:902] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f: Initiating tablet copy from peer 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:40.557507 8923 tablet_copy_client.cc:308] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33439
I20240503 13:59:40.560274 6929 tablet_copy_service.cc:140] P 52a58bb8fe574f5ab1485c59bf994570: Received BeginTabletCopySession request for tablet ef2d39830da24221a3dc44da57c0a6e8 from peer 083b1c5c79a04af899eea33ba9f4502f ({username='slave'} at 127.0.99.134:42783)
I20240503 13:59:40.560356 6929 tablet_copy_service.cc:161] P 52a58bb8fe574f5ab1485c59bf994570: Beginning new tablet copy session on tablet ef2d39830da24221a3dc44da57c0a6e8 from peer 083b1c5c79a04af899eea33ba9f4502f at {username='slave'} at 127.0.99.134:42783: session id = 083b1c5c79a04af899eea33ba9f4502f-ef2d39830da24221a3dc44da57c0a6e8
I20240503 13:59:40.561167 6929 tablet_copy_source_session.cc:205] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:40.565748 8923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2d39830da24221a3dc44da57c0a6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:40.570485 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.570729 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.571686 8923 tablet_copy_client.cc:783] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:40.571841 8923 tablet_copy_client.cc:647] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:40.576100 8923 tablet_copy_client.cc:523] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:40.577884 8923 tablet_bootstrap.cc:492] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap starting.
I20240503 13:59:40.620337 8287 raft_consensus.cc:1203] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.79->[1.80-1.107] Dedup: 1.91->[1.92-1.107]
I20240503 13:59:40.631158 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.645201 8022 raft_consensus.cc:1203] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.93->[1.94-1.124] Dedup: 1.108->[1.109-1.124]
I20240503 13:59:40.725533 8757 tablet_bootstrap.cc:492] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap replayed 1/1 log segments. Stats: ops{read=103 overwritten=0 applied=103 ignored=0} inserts{seen=788 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.726022 8757 tablet_bootstrap.cc:492] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847: Bootstrap complete.
I20240503 13:59:40.726188 8757 ts_tablet_manager.cc:1367] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847: Time spent bootstrapping tablet: real 0.516s user 0.068s sys 0.025s
I20240503 13:59:40.726549 8757 raft_consensus.cc:348] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:40.726660 8757 raft_consensus.cc:729] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f82135907754dd8b69d08d6b7eae847, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.726783 8757 consensus_queue.cc:260] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 103, Last appended: 1.103, Last appended by leader: 103, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:40.727052 8757 ts_tablet_manager.cc:1398] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:40.727674 6929 tablet_copy_service.cc:342] P 52a58bb8fe574f5ab1485c59bf994570: Request end of tablet copy session 4f82135907754dd8b69d08d6b7eae847-4f882dbedd134f2998642ac747567db0 received from {username='slave'} at 127.0.99.135:45715
I20240503 13:59:40.727757 6929 tablet_copy_service.cc:433] P 52a58bb8fe574f5ab1485c59bf994570: ending tablet copy session 4f82135907754dd8b69d08d6b7eae847-4f882dbedd134f2998642ac747567db0 on tablet 4f882dbedd134f2998642ac747567db0 with peer 4f82135907754dd8b69d08d6b7eae847
I20240503 13:59:40.784839 8419 raft_consensus.cc:1203] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.91->[1.92-1.120] Dedup: 1.103->[1.104-1.120]
I20240503 13:59:40.786676 8923 tablet_bootstrap.cc:492] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap replayed 1/1 log segments. Stats: ops{read=98 overwritten=0 applied=97 ignored=0} inserts{seen=769 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 13:59:40.786962 8923 tablet_bootstrap.cc:492] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f: Bootstrap complete.
I20240503 13:59:40.787148 8923 ts_tablet_manager.cc:1367] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f: Time spent bootstrapping tablet: real 0.209s user 0.077s sys 0.008s
I20240503 13:59:40.787559 8923 raft_consensus.cc:348] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:40.787815 8923 raft_consensus.cc:729] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083b1c5c79a04af899eea33ba9f4502f, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.787940 8923 consensus_queue.cc:260] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 1.98, Last appended by leader: 98, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:40.788200 8923 ts_tablet_manager.cc:1398] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 13:59:40.794587 6929 tablet_copy_service.cc:342] P 52a58bb8fe574f5ab1485c59bf994570: Request end of tablet copy session 083b1c5c79a04af899eea33ba9f4502f-ef2d39830da24221a3dc44da57c0a6e8 received from {username='slave'} at 127.0.99.134:42783
I20240503 13:59:40.794664 6929 tablet_copy_service.cc:433] P 52a58bb8fe574f5ab1485c59bf994570: ending tablet copy session 083b1c5c79a04af899eea33ba9f4502f-ef2d39830da24221a3dc44da57c0a6e8 on tablet ef2d39830da24221a3dc44da57c0a6e8 with peer 083b1c5c79a04af899eea33ba9f4502f
I20240503 13:59:40.878135 8793 tablet_bootstrap.cc:492] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap replayed 1/1 log segments. Stats: ops{read=104 overwritten=0 applied=104 ignored=0} inserts{seen=789 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:40.878679 8793 tablet_bootstrap.cc:492] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384: Bootstrap complete.
I20240503 13:59:40.878918 8793 ts_tablet_manager.cc:1367] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384: Time spent bootstrapping tablet: real 0.441s user 0.087s sys 0.008s
I20240503 13:59:40.879447 8793 raft_consensus.cc:348] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:40.879565 8793 raft_consensus.cc:729] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5b7c28dd2a4920a2b4e97d1afe2384, State: Initialized, Role: FOLLOWER
I20240503 13:59:40.879843 8793 consensus_queue.cc:260] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 104, Last appended: 1.104, Last appended by leader: 104, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:40.882050 8793 ts_tablet_manager.cc:1398] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 13:59:40.891957 6798 tablet_copy_service.cc:342] P 791cddb67b7e415aad7edb5151037418: Request end of tablet copy session 4c5b7c28dd2a4920a2b4e97d1afe2384-a82b94a4baa84d778cc280d58c18af5b received from {username='slave'} at 127.0.99.133:34371
I20240503 13:59:40.892072 6798 tablet_copy_service.cc:433] P 791cddb67b7e415aad7edb5151037418: ending tablet copy session 4c5b7c28dd2a4920a2b4e97d1afe2384-a82b94a4baa84d778cc280d58c18af5b on tablet a82b94a4baa84d778cc280d58c18af5b with peer 4c5b7c28dd2a4920a2b4e97d1afe2384
I20240503 13:59:40.920928 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:40.923903 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.935951 6758 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:40.945485 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.948022 6889 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:40.951867 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:40.957787 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:40.978596 8155 raft_consensus.cc:1203] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.92->[1.93-1.106] Dedup: 1.104->[1.105-1.106]
I20240503 13:59:41.086350 8286 raft_consensus.cc:1203] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.88->[1.89-1.103] Dedup: 1.98->[1.99-1.103]
I20240503 13:59:41.233839 6906 consensus_queue.cc:237] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 1.123, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:41.237156 8288 raft_consensus.cc:1261] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240503 13:59:41.238101 8548 consensus_queue.cc:1035] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240503 13:59:41.244400 8548 raft_consensus.cc:2932] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.124: config changed from index 80 to 124, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:41.250125 8286 raft_consensus.cc:2932] T 70583380c4e24b4694fc5477d880d944 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 80 to 124, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:41.252620 6636 catalog_manager.cc:5484] T 70583380c4e24b4694fc5477d880d944 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: config changed from index 80 to 124, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:41.394075 6758 tablet_service.cc:1514] Processing DeleteTablet for tablet 70583380c4e24b4694fc5477d880d944 with delete_type TABLET_DATA_TOMBSTONED (TS 791cddb67b7e415aad7edb5151037418 not found in new config with opid_index 124) from {username='slave'} at 127.0.0.1:40392
I20240503 13:59:41.405167 9163 tablet_replica.cc:330] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418: stopping tablet replica
I20240503 13:59:41.405489 9163 raft_consensus.cc:2229] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.405666 9163 pending_rounds.cc:62] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418: Trying to abort 1 pending ops.
I20240503 13:59:41.405727 9163 pending_rounds.cc:68] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418: Aborting op as it isn't in flight: id { term: 1 index: 123 } timestamp: 7023594623895896064 op_type: WRITE_OP write_request { tablet_id: "70583380c4e24b4694fc5477d880d944" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000G\3244Ve\317\204\016\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""b92d835db61719a4" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023594623868596224 authz_token { token_data: "\010\365\335\323\261\006\"3\n\005slave\022*\n 1b4f4315b2d34ad8a7a1bdd75ec294ef\020\001\030\001 \001(\001" signature: "<redacted>""\035\342U\211\374\326\225\360Jv@y\201\2134\361\367v\327:\2641{\004(\241E\n\267\030\031\250\342\325S{\3272^\205\034V3R\245\035\363[^\345\0260\022\007_9\300\010\003i\266\347kK" signing_key_seq_num: 0 } } request_id { client_id: "12983e91c3d648edbbef278864ab2d9e" seq_no: 336 first_incomplete_seq_no: 336 attempt_no: 0 }
I20240503 13:59:41.406014 9163 raft_consensus.cc:2258] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.412853 9163 ts_tablet_manager.cc:1875] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.420279 9163 ts_tablet_manager.cc:1888] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.123
I20240503 13:59:41.420360 9163 log.cc:1196] T 70583380c4e24b4694fc5477d880d944 P 791cddb67b7e415aad7edb5151037418: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/70583380c4e24b4694fc5477d880d944
I20240503 13:59:41.421593 6620 catalog_manager.cc:4830] TS 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893): tablet 70583380c4e24b4694fc5477d880d944 (table rebalance_test_table_4 [id=1b4f4315b2d34ad8a7a1bdd75ec294ef]) successfully deleted
I20240503 13:59:41.479033 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:41.504025 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:41.519343 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.527817 6758 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:41.567386 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.577991 6889 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:41.632838 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:41.913533 7039 consensus_queue.cc:237] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 175, Committed index: 175, Last appended: 1.175, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:41.914690 8023 raft_consensus.cc:1261] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 1 index: 176. (index mismatch)
I20240503 13:59:41.915259 8680 consensus_queue.cc:1035] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 175, Time since last communication: 0.000s
I20240503 13:59:41.918027 8557 raft_consensus.cc:2932] T e026fe8a768941ce84378f23b3dabff2 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Committing config change with OpId 1.176: config changed from index 94 to 176, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:41.920886 8022 raft_consensus.cc:2932] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 94 to 176, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:41.931039 6632 catalog_manager.cc:5484] T e026fe8a768941ce84378f23b3dabff2 P 40e8eacd02b44bb8bbec0a1dc205a671 reported cstate change: config changed from index 94 to 176, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:41.953081 6889 tablet_service.cc:1514] Processing DeleteTablet for tablet e026fe8a768941ce84378f23b3dabff2 with delete_type TABLET_DATA_TOMBSTONED (TS 52a58bb8fe574f5ab1485c59bf994570 not found in new config with opid_index 176) from {username='slave'} at 127.0.0.1:36310
I20240503 13:59:41.955296 9231 tablet_replica.cc:330] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570: stopping tablet replica
I20240503 13:59:41.955915 9231 raft_consensus.cc:2229] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:41.956169 9231 raft_consensus.cc:2258] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:41.966096 9231 ts_tablet_manager.cc:1875] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:41.969664 9231 ts_tablet_manager.cc:1888] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.175
I20240503 13:59:41.969745 9231 log.cc:1196] T e026fe8a768941ce84378f23b3dabff2 P 52a58bb8fe574f5ab1485c59bf994570: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e026fe8a768941ce84378f23b3dabff2
I20240503 13:59:41.990833 6620 catalog_manager.cc:4830] TS 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439): tablet e026fe8a768941ce84378f23b3dabff2 (table rebalance_test_table_3 [id=bffd930039c544a68647ac9a405cf590]) successfully deleted
I20240503 13:59:42.382968 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:42.389369 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.399016 6758 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:42.406811 6889 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:42.443451 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.460219 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:42.499382 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:42.652823 6906 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2b880960ed44b20b81ddf91c8534638"
dest_uuid: "52a58bb8fe574f5ab1485c59bf994570"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60832
I20240503 13:59:42.653000 6906 raft_consensus.cc:595] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:42.659343 8548 raft_consensus.cc:979] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570: : Instructing follower 14c195eab0b34945afd10f3291ed080b to start an election
I20240503 13:59:42.659456 8548 raft_consensus.cc:1067] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Signalling peer 14c195eab0b34945afd10f3291ed080b to start an election
I20240503 13:59:42.668982 7036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2b880960ed44b20b81ddf91c8534638"
dest_uuid: "14c195eab0b34945afd10f3291ed080b"
from {username='slave'} at 127.0.99.130:37511
I20240503 13:59:42.673645 7036 raft_consensus.cc:482] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:42.673825 7036 raft_consensus.cc:3037] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:59:42.681080 6869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36426: Service unavailable: leader transfer in progress
I20240503 13:59:42.682276 7036 raft_consensus.cc:504] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
W20240503 13:59:42.683732 6869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36426: Service unavailable: leader transfer in progress
I20240503 13:59:42.687597 7036 leader_election.cc:288] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 2 election: Requested vote from peers 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439), 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133:39967)
I20240503 13:59:42.689407 6906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b880960ed44b20b81ddf91c8534638" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 2 candidate_status { last_received { term: 1 index: 185 } } ignore_live_leader: true dest_uuid: "52a58bb8fe574f5ab1485c59bf994570"
I20240503 13:59:42.689740 6906 raft_consensus.cc:3032] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:42.689814 6906 raft_consensus.cc:729] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Running, Role: LEADER
I20240503 13:59:42.689965 6906 consensus_queue.cc:260] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 185, Committed index: 185, Last appended: 1.185, Last appended by leader: 185, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:42.690383 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b880960ed44b20b81ddf91c8534638" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 2 candidate_status { last_received { term: 1 index: 185 } } ignore_live_leader: true dest_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384"
I20240503 13:59:42.690630 6906 raft_consensus.cc:3037] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:42.690768 8156 raft_consensus.cc:3037] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:42.692078 6906 raft_consensus.cc:2454] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 2.
I20240503 13:59:42.693696 8156 raft_consensus.cc:2454] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 2.
I20240503 13:59:42.692489 6973 leader_election.cc:302] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 52a58bb8fe574f5ab1485c59bf994570; no voters:
W20240503 13:59:42.694375 6869 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36426: Illegal state: replica 52a58bb8fe574f5ab1485c59bf994570 is not leader of this config: current role FOLLOWER
I20240503 13:59:42.694975 8557 raft_consensus.cc:2781] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:42.695068 8557 raft_consensus.cc:686] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 2 LEADER]: Becoming Leader. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Running, Role: LEADER
I20240503 13:59:42.695185 8557 consensus_queue.cc:237] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 185, Committed index: 185, Last appended: 1.185, Last appended by leader: 185, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:42.701891 6635 catalog_manager.cc:5484] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b reported cstate change: term changed from 1 to 2, leader changed from 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) to 14c195eab0b34945afd10f3291ed080b (127.0.99.131). New cstate: current_term: 2 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:42.778403 6909 raft_consensus.cc:1261] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 2 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 185. Preceding OpId from leader: term: 2 index: 187. (index mismatch)
I20240503 13:59:42.779013 7819 consensus_queue.cc:1035] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 185, Time since last communication: 0.000s
I20240503 13:59:42.779064 8156 raft_consensus.cc:1261] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 2 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 185. Preceding OpId from leader: term: 2 index: 187. (index mismatch)
I20240503 13:59:42.782800 8552 consensus_queue.cc:1035] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 185, Time since last communication: 0.000s
I20240503 13:59:43.085897 7020 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:43.139325 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.156908 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.171648 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.195777 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:43.210179 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:43.289959 6758 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:43.481148 7036 consensus_queue.cc:237] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 185, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:43.495163 8154 raft_consensus.cc:1261] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 2 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20240503 13:59:43.497898 8552 consensus_queue.cc:1035] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20240503 13:59:43.500614 8552 raft_consensus.cc:2932] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index 82 to 207, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:43.527725 8154 raft_consensus.cc:2932] T f2b880960ed44b20b81ddf91c8534638 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index 82 to 207, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:43.530799 6634 catalog_manager.cc:5484] T f2b880960ed44b20b81ddf91c8534638 P 14c195eab0b34945afd10f3291ed080b reported cstate change: config changed from index 82 to 207, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:43.554260 6889 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b880960ed44b20b81ddf91c8534638 with delete_type TABLET_DATA_TOMBSTONED (TS 52a58bb8fe574f5ab1485c59bf994570 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:36310
I20240503 13:59:43.575820 9363 tablet_replica.cc:330] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570: stopping tablet replica
I20240503 13:59:43.576223 9363 raft_consensus.cc:2229] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:43.576648 9363 raft_consensus.cc:2258] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:43.577682 9363 ts_tablet_manager.cc:1875] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:43.580862 9363 ts_tablet_manager.cc:1888] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20240503 13:59:43.581012 9363 log.cc:1196] T f2b880960ed44b20b81ddf91c8534638 P 52a58bb8fe574f5ab1485c59bf994570: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f2b880960ed44b20b81ddf91c8534638
I20240503 13:59:43.584544 6620 catalog_manager.cc:4830] TS 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439): tablet f2b880960ed44b20b81ddf91c8534638 (table rebalance_test_table_1 [id=bb88829318084b1e8ec739465823de3c]) successfully deleted
I20240503 13:59:43.848636 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:43.858386 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.861424 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.874316 7020 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:43.891780 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:43.896637 6758 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:43.960439 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:44.039788 6776 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a871746624e442597588274959d5e24"
dest_uuid: "791cddb67b7e415aad7edb5151037418"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34264
I20240503 13:59:44.039971 6776 raft_consensus.cc:595] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:44.043599 9000 raft_consensus.cc:979] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: : Instructing follower 14c195eab0b34945afd10f3291ed080b to start an election
I20240503 13:59:44.043705 9000 raft_consensus.cc:1067] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Signalling peer 14c195eab0b34945afd10f3291ed080b to start an election
I20240503 13:59:44.044539 7039 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0a871746624e442597588274959d5e24"
dest_uuid: "14c195eab0b34945afd10f3291ed080b"
from {username='slave'} at 127.0.99.129:39399
I20240503 13:59:44.044668 7039 raft_consensus.cc:482] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:44.044718 7039 raft_consensus.cc:3037] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:44.045712 7039 raft_consensus.cc:504] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:44.046133 7039 leader_election.cc:288] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 2 election: Requested vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893), 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579)
I20240503 13:59:44.047036 6776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a871746624e442597588274959d5e24" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 2 candidate_status { last_received { term: 1 index: 216 } } ignore_live_leader: true dest_uuid: "791cddb67b7e415aad7edb5151037418"
I20240503 13:59:44.047140 6776 raft_consensus.cc:3032] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:44.047180 6776 raft_consensus.cc:729] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 791cddb67b7e415aad7edb5151037418, State: Running, Role: LEADER
I20240503 13:59:44.047345 6776 consensus_queue.cc:260] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 1.216, Last appended by leader: 216, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:44.047467 6776 raft_consensus.cc:3037] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:44.048493 6776 raft_consensus.cc:2454] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 2.
I20240503 13:59:44.050662 6973 leader_election.cc:302] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:44.051123 9233 raft_consensus.cc:2781] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:44.051223 9233 raft_consensus.cc:686] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 2 LEADER]: Becoming Leader. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Running, Role: LEADER
I20240503 13:59:44.051379 9233 consensus_queue.cc:237] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 1.216, Last appended by leader: 216, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:44.054483 6634 catalog_manager.cc:5484] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b reported cstate change: term changed from 1 to 2, leader changed from 791cddb67b7e415aad7edb5151037418 (127.0.99.129) to 14c195eab0b34945afd10f3291ed080b (127.0.99.131). New cstate: current_term: 2 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
W20240503 13:59:44.057543 6735 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40450: Illegal state: replica 791cddb67b7e415aad7edb5151037418 is not leader of this config: current role FOLLOWER
I20240503 13:59:44.067394 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a871746624e442597588274959d5e24" candidate_uuid: "14c195eab0b34945afd10f3291ed080b" candidate_term: 2 candidate_status { last_received { term: 1 index: 216 } } ignore_live_leader: true dest_uuid: "083b1c5c79a04af899eea33ba9f4502f"
I20240503 13:59:44.067543 8287 raft_consensus.cc:3037] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:44.068658 8287 raft_consensus.cc:2454] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c195eab0b34945afd10f3291ed080b in term 2.
I20240503 13:59:44.104099 6776 raft_consensus.cc:1261] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 2 index: 218. (index mismatch)
I20240503 13:59:44.104218 8287 raft_consensus.cc:1261] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f [term 2 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 2 index: 218. (index mismatch)
I20240503 13:59:44.105502 9329 consensus_queue.cc:1035] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 215, Time since last communication: 0.000s
I20240503 13:59:44.105717 8557 consensus_queue.cc:1035] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20240503 13:59:44.110075 8556 mvcc.cc:205] Tried to move back new op lower bound from 7023594635684941824 to 7023594635475963904. Current Snapshot: MvccSnapshot[applied={T|T < 7023594635684941824}]
I20240503 13:59:44.115049 9375 mvcc.cc:205] Tried to move back new op lower bound from 7023594635684941824 to 7023594635475963904. Current Snapshot: MvccSnapshot[applied={T|T < 7023594635684941824}]
I20240503 13:59:44.400872 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:44.423913 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:44.426132 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.432740 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.437218 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:44.460000 7020 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:44.518757 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:44.762313 7038 consensus_queue.cc:237] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 2.234, Last appended by leader: 216, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:44.765138 8287 raft_consensus.cc:1261] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f [term 2 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 2 index: 234. Preceding OpId from leader: term: 2 index: 235. (index mismatch)
I20240503 13:59:44.766512 7819 consensus_queue.cc:1035] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240503 13:59:44.769246 7819 raft_consensus.cc:2932] T 0a871746624e442597588274959d5e24 P 14c195eab0b34945afd10f3291ed080b [term 2 LEADER]: Committing config change with OpId 2.235: config changed from index 74 to 235, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:44.769870 8287 raft_consensus.cc:2932] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f [term 2 FOLLOWER]: Committing config change with OpId 2.235: config changed from index 74 to 235, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:44.772836 6635 catalog_manager.cc:5484] T 0a871746624e442597588274959d5e24 P 083b1c5c79a04af899eea33ba9f4502f reported cstate change: config changed from index 74 to 235, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:44.798353 6758 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a871746624e442597588274959d5e24 with delete_type TABLET_DATA_TOMBSTONED (TS 791cddb67b7e415aad7edb5151037418 not found in new config with opid_index 235) from {username='slave'} at 127.0.0.1:40392
I20240503 13:59:44.801705 9486 tablet_replica.cc:330] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: stopping tablet replica
I20240503 13:59:44.802018 9486 raft_consensus.cc:2229] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:44.806284 9486 pending_rounds.cc:62] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: Trying to abort 1 pending ops.
I20240503 13:59:44.806411 9486 pending_rounds.cc:68] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: Aborting op as it isn't in flight: id { term: 2 index: 234 } timestamp: 7023594638336786432 op_type: WRITE_OP write_request { tablet_id: "0a871746624e442597588274959d5e24" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\305x+a\372c\277\244\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""965297e010e1fd26" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023594638296481792 authz_token { token_data: "\010\365\335\323\261\006\"3\n\005slave\022*\n f9d13cbf321a4a76b3f59808a6da0969\020\001\030\001 \001(\001" signature: "<redacted>""\010Uj\232\3634\242\007_\245\345=$\332\367\233\2370\304\236u\261\017w\274\233\366\202\371e\222\364\277\216\004\331\276\2500\036\322\205\2701O1Y\r\214\0333\372\372\211w\024\314\027b\017\027\347L\245" signing_key_seq_num: 0 } } request_id { client_id: "0b6b0c85b7c7435ca2aac3fd00509474" seq_no: 646 first_incomplete_seq_no: 646 attempt_no: 0 }
I20240503 13:59:44.806891 9486 raft_consensus.cc:2258] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:44.808405 9486 ts_tablet_manager.cc:1875] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:44.811829 9486 ts_tablet_manager.cc:1888] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.234
I20240503 13:59:44.811972 9486 log.cc:1196] T 0a871746624e442597588274959d5e24 P 791cddb67b7e415aad7edb5151037418: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0a871746624e442597588274959d5e24
I20240503 13:59:44.816089 6620 catalog_manager.cc:4830] TS 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893): tablet 0a871746624e442597588274959d5e24 (table rebalance_test_table_0 [id=f9d13cbf321a4a76b3f59808a6da0969]) successfully deleted
I20240503 13:59:45.154064 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:45.154691 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:45.172308 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:45.173705 7020 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:45.222895 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:45.267408 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:45.286267 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:45.466861 6777 consensus_queue.cc:237] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 260, Committed index: 260, Last appended: 1.260, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:45.468035 8418 raft_consensus.cc:1261] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Refusing update from remote peer 791cddb67b7e415aad7edb5151037418: Log matching property violated. Preceding OpId in replica: term: 1 index: 260. Preceding OpId from leader: term: 1 index: 261. (index mismatch)
I20240503 13:59:45.468523 9154 consensus_queue.cc:1035] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 261, Last known committed idx: 260, Time since last communication: 0.000s
I20240503 13:59:45.470161 9154 raft_consensus.cc:2932] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Committing config change with OpId 1.261: config changed from index 86 to 261, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:45.473351 6633 catalog_manager.cc:5484] T 5fb5bbc56575455eb122d1e401fbdc64 P 791cddb67b7e415aad7edb5151037418 reported cstate change: config changed from index 86 to 261, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "791cddb67b7e415aad7edb5151037418" committed_config { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:45.473652 8420 raft_consensus.cc:2932] T 5fb5bbc56575455eb122d1e401fbdc64 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Committing config change with OpId 1.261: config changed from index 86 to 261, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 261 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:45.536185 7020 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb5bbc56575455eb122d1e401fbdc64 with delete_type TABLET_DATA_TOMBSTONED (TS 14c195eab0b34945afd10f3291ed080b not found in new config with opid_index 261) from {username='slave'} at 127.0.0.1:42338
I20240503 13:59:45.581828 9558 tablet_replica.cc:330] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b: stopping tablet replica
I20240503 13:59:45.583107 9558 raft_consensus.cc:2229] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:45.583352 9558 raft_consensus.cc:2258] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:45.591554 9558 ts_tablet_manager.cc:1875] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:45.618640 9558 ts_tablet_manager.cc:1888] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.260
I20240503 13:59:45.618840 9558 log.cc:1196] T 5fb5bbc56575455eb122d1e401fbdc64 P 14c195eab0b34945afd10f3291ed080b: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5fb5bbc56575455eb122d1e401fbdc64
I20240503 13:59:45.620013 6620 catalog_manager.cc:4830] TS 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699): tablet 5fb5bbc56575455eb122d1e401fbdc64 (table rebalance_test_table_1 [id=bb88829318084b1e8ec739465823de3c]) successfully deleted
I20240503 13:59:45.832880 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:45.844987 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:45.861804 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:45.863194 7020 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:45.867513 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:45.881652 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:45.892572 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:46.120002 6909 consensus_queue.cc:237] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 261, Committed index: 261, Last appended: 1.261, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:46.121381 8023 raft_consensus.cc:1261] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 261. Preceding OpId from leader: term: 1 index: 262. (index mismatch)
I20240503 13:59:46.121703 9198 consensus_queue.cc:1035] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 262, Last known committed idx: 261, Time since last communication: 0.000s
I20240503 13:59:46.126257 9198 raft_consensus.cc:2932] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.262: config changed from index 70 to 262, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:46.126761 8021 raft_consensus.cc:2932] T 6409f411c5d24e97b34aa9cce7c5a5ea P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Committing config change with OpId 1.262: config changed from index 70 to 262, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:46.134189 6633 catalog_manager.cc:5484] T 6409f411c5d24e97b34aa9cce7c5a5ea P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: config changed from index 70 to 262, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 262 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:46.278764 7020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6409f411c5d24e97b34aa9cce7c5a5ea with delete_type TABLET_DATA_TOMBSTONED (TS 14c195eab0b34945afd10f3291ed080b not found in new config with opid_index 262) from {username='slave'} at 127.0.0.1:42338
I20240503 13:59:46.279068 9634 tablet_replica.cc:330] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b: stopping tablet replica
I20240503 13:59:46.280071 9634 raft_consensus.cc:2229] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:46.283090 9634 raft_consensus.cc:2258] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:46.284031 9634 ts_tablet_manager.cc:1875] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:46.290306 9634 ts_tablet_manager.cc:1888] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.261
I20240503 13:59:46.290383 9634 log.cc:1196] T 6409f411c5d24e97b34aa9cce7c5a5ea P 14c195eab0b34945afd10f3291ed080b: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6409f411c5d24e97b34aa9cce7c5a5ea
I20240503 13:59:46.293323 6620 catalog_manager.cc:4830] TS 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699): tablet 6409f411c5d24e97b34aa9cce7c5a5ea (table rebalance_test_table_0 [id=f9d13cbf321a4a76b3f59808a6da0969]) successfully deleted
I20240503 13:59:46.396194 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:46.421239 7020 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:46.440984 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:46.447508 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:46.461529 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:46.476805 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:46.487627 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:46.755236 6778 consensus_queue.cc:237] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 1.273, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:46.760231 8153 raft_consensus.cc:1261] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Refusing update from remote peer 791cddb67b7e415aad7edb5151037418: Log matching property violated. Preceding OpId in replica: term: 1 index: 273. Preceding OpId from leader: term: 1 index: 275. (index mismatch)
I20240503 13:59:46.760659 9154 consensus_queue.cc:1035] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 274, Last known committed idx: 273, Time since last communication: 0.000s
I20240503 13:59:46.763450 8606 raft_consensus.cc:2932] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Committing config change with OpId 1.274: config changed from index 93 to 274, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:46.766258 6635 catalog_manager.cc:5484] T a82b94a4baa84d778cc280d58c18af5b P 791cddb67b7e415aad7edb5151037418 reported cstate change: config changed from index 93 to 274, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "791cddb67b7e415aad7edb5151037418" committed_config { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:46.766937 8153 raft_consensus.cc:2932] T a82b94a4baa84d778cc280d58c18af5b P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Committing config change with OpId 1.274: config changed from index 93 to 274, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 274 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:46.819847 6889 tablet_service.cc:1514] Processing DeleteTablet for tablet a82b94a4baa84d778cc280d58c18af5b with delete_type TABLET_DATA_TOMBSTONED (TS 52a58bb8fe574f5ab1485c59bf994570 not found in new config with opid_index 274) from {username='slave'} at 127.0.0.1:36310
I20240503 13:59:46.843603 9703 tablet_replica.cc:330] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570: stopping tablet replica
I20240503 13:59:46.843922 9703 raft_consensus.cc:2229] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:46.844110 9703 raft_consensus.cc:2258] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:46.846078 9703 ts_tablet_manager.cc:1875] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:46.851989 9703 ts_tablet_manager.cc:1888] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.273
I20240503 13:59:46.852154 9703 log.cc:1196] T a82b94a4baa84d778cc280d58c18af5b P 52a58bb8fe574f5ab1485c59bf994570: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a82b94a4baa84d778cc280d58c18af5b
I20240503 13:59:46.853422 6620 catalog_manager.cc:4830] TS 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439): tablet a82b94a4baa84d778cc280d58c18af5b (table rebalance_test_table_2 [id=04d1a2edbd1a48bc84eae4ce2c7379ff]) successfully deleted
I20240503 13:59:47.187765 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:47.196185 8136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:47.235129 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:47.248772 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:47.264544 7020 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 13:59:47.269836 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:47.301052 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:47.501047 7039 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57ea09e3f1f74d74839ddb0afd90acc2"
dest_uuid: "14c195eab0b34945afd10f3291ed080b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59114
I20240503 13:59:47.501242 7039 raft_consensus.cc:595] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:47.503587 8552 raft_consensus.cc:979] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: : Instructing follower 4c5b7c28dd2a4920a2b4e97d1afe2384 to start an election
I20240503 13:59:47.503840 7819 raft_consensus.cc:1067] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Signalling peer 4c5b7c28dd2a4920a2b4e97d1afe2384 to start an election
I20240503 13:59:47.508112 8154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57ea09e3f1f74d74839ddb0afd90acc2"
dest_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384"
from {username='slave'} at 127.0.99.131:60973
I20240503 13:59:47.509057 8154 raft_consensus.cc:482] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:47.509133 8154 raft_consensus.cc:3037] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:47.510329 8154 raft_consensus.cc:504] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:47.512619 7039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ea09e3f1f74d74839ddb0afd90acc2" candidate_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" candidate_term: 2 candidate_status { last_received { term: 1 index: 353 } } ignore_live_leader: true dest_uuid: "14c195eab0b34945afd10f3291ed080b"
I20240503 13:59:47.512787 7039 raft_consensus.cc:3032] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:47.512874 7039 raft_consensus.cc:729] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Running, Role: LEADER
I20240503 13:59:47.513080 7039 consensus_queue.cc:260] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 1.353, Last appended by leader: 353, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:47.514143 6776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ea09e3f1f74d74839ddb0afd90acc2" candidate_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" candidate_term: 2 candidate_status { last_received { term: 1 index: 353 } } ignore_live_leader: true dest_uuid: "791cddb67b7e415aad7edb5151037418"
I20240503 13:59:47.514312 6776 raft_consensus.cc:3037] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:47.516001 6776 raft_consensus.cc:2454] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c5b7c28dd2a4920a2b4e97d1afe2384 in term 2.
I20240503 13:59:47.517699 8154 leader_election.cc:288] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [CANDIDATE]: Term 2 election: Requested vote from peers 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893), 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:47.518337 7039 raft_consensus.cc:3037] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:59:47.519737 7000 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42404: Illegal state: replica 14c195eab0b34945afd10f3291ed080b is not leader of this config: current role FOLLOWER
I20240503 13:59:47.520587 7039 raft_consensus.cc:2454] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c5b7c28dd2a4920a2b4e97d1afe2384 in term 2.
I20240503 13:59:47.519130 8089 leader_election.cc:302] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c5b7c28dd2a4920a2b4e97d1afe2384, 791cddb67b7e415aad7edb5151037418; no voters:
I20240503 13:59:47.533829 9754 raft_consensus.cc:2781] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:47.535108 9754 raft_consensus.cc:686] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 2 LEADER]: Becoming Leader. State: Replica: 4c5b7c28dd2a4920a2b4e97d1afe2384, State: Running, Role: LEADER
I20240503 13:59:47.539553 9754 consensus_queue.cc:237] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 1.353, Last appended by leader: 353, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:47.542567 6631 catalog_manager.cc:5484] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 reported cstate change: term changed from 1 to 2, leader changed from 14c195eab0b34945afd10f3291ed080b (127.0.99.131) to 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
W20240503 13:59:47.575819 6737 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34378: Illegal state: replica 791cddb67b7e415aad7edb5151037418 is not leader of this config: current role FOLLOWER
I20240503 13:59:47.591600 7038 raft_consensus.cc:1261] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Refusing update from remote peer 4c5b7c28dd2a4920a2b4e97d1afe2384: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 2 index: 355. (index mismatch)
I20240503 13:59:47.594012 9756 consensus_queue.cc:1035] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240503 13:59:47.595212 6776 raft_consensus.cc:1261] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Refusing update from remote peer 4c5b7c28dd2a4920a2b4e97d1afe2384: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 2 index: 355. (index mismatch)
I20240503 13:59:47.596930 9756 consensus_queue.cc:1035] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240503 13:59:47.742094 7020 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:47.744962 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:47.764580 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:47.786459 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:47.789767 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 13:59:47.797793 6711 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.133:39967, user_credentials={real_user=slave}} blocked reactor thread for 48275.7us
I20240503 13:59:47.808279 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:47.836334 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:48.054693 8154 consensus_queue.cc:237] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 2.370, Last appended by leader: 353, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:48.066355 6776 raft_consensus.cc:1261] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Refusing update from remote peer 4c5b7c28dd2a4920a2b4e97d1afe2384: Log matching property violated. Preceding OpId in replica: term: 2 index: 369. Preceding OpId from leader: term: 2 index: 371. (index mismatch)
I20240503 13:59:48.067174 9771 consensus_queue.cc:1035] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 369, Time since last communication: 0.000s
I20240503 13:59:48.074658 9771 raft_consensus.cc:2932] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 2 LEADER]: Committing config change with OpId 2.371: config changed from index 61 to 371, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:48.078380 6631 catalog_manager.cc:5484] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 reported cstate change: config changed from index 61 to 371, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" committed_config { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:48.084538 6775 raft_consensus.cc:2932] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Committing config change with OpId 2.371: config changed from index 61 to 371, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:48.102346 7020 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ea09e3f1f74d74839ddb0afd90acc2 with delete_type TABLET_DATA_TOMBSTONED (TS 14c195eab0b34945afd10f3291ed080b not found in new config with opid_index 371) from {username='slave'} at 127.0.0.1:42338
I20240503 13:59:48.187554 9829 tablet_replica.cc:330] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: stopping tablet replica
I20240503 13:59:48.192092 9829 raft_consensus.cc:2229] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:48.192276 9829 pending_rounds.cc:62] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: Trying to abort 1 pending ops.
I20240503 13:59:48.192406 9829 pending_rounds.cc:68] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: Aborting op as it isn't in flight: id { term: 2 index: 370 } timestamp: 7023594651857805312 op_type: WRITE_OP write_request { tablet_id: "57ea09e3f1f74d74839ddb0afd90acc2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\275T\232Y\315\331C\003\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""2a023713eedbed88" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023594651820195840 authz_token { token_data: "\010\365\335\323\261\006\"3\n\005slave\022*\n bffd930039c544a68647ac9a405cf590\020\001\030\001 \001(\001" signature: "<redacted>""\305\233C%\221\241f\233\364[\014\262F3!\025Z\344o\344\251S\355\376\177\227\236%=zdR\304\020\222\035>\277\376\363\236_?\210I\304\003L\305\300\032_s\344\256C\330\346\222J\372 =\020" signing_key_seq_num: 0 } } request_id { client_id: "d3d84140f9384e439b2d83449f2e5f65" seq_no: 1069 first_incomplete_seq_no: 1069 attempt_no: 0 }
I20240503 13:59:48.193019 9829 raft_consensus.cc:2258] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:48.193812 9829 ts_tablet_manager.cc:1875] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:48.202955 9829 ts_tablet_manager.cc:1888] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.370
I20240503 13:59:48.203125 9829 log.cc:1196] T 57ea09e3f1f74d74839ddb0afd90acc2 P 14c195eab0b34945afd10f3291ed080b: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/57ea09e3f1f74d74839ddb0afd90acc2
I20240503 13:59:48.203912 6620 catalog_manager.cc:4830] TS 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699): tablet 57ea09e3f1f74d74839ddb0afd90acc2 (table rebalance_test_table_3 [id=bffd930039c544a68647ac9a405cf590]) successfully deleted
I20240503 13:59:48.506109 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:48.524425 7020 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:48.528404 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:48.547740 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:48.557176 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:48.558931 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:48.593058 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:48.769686 6776 consensus_queue.cc:237] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 1.301, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } }
I20240503 13:59:48.771374 8153 raft_consensus.cc:1261] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Refusing update from remote peer 791cddb67b7e415aad7edb5151037418: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20240503 13:59:48.772490 9154 consensus_queue.cc:1035] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20240503 13:59:48.776623 9154 raft_consensus.cc:2932] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 [term 1 LEADER]: Committing config change with OpId 1.302: config changed from index 63 to 302, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:48.777016 8153 raft_consensus.cc:2932] T 527c7e7cf4f24ce3bed276c1860aa041 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 63 to 302, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:48.779579 6631 catalog_manager.cc:5484] T 527c7e7cf4f24ce3bed276c1860aa041 P 791cddb67b7e415aad7edb5151037418 reported cstate change: config changed from index 63 to 302, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "791cddb67b7e415aad7edb5151037418" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } }
I20240503 13:59:48.844678 6889 tablet_service.cc:1514] Processing DeleteTablet for tablet 527c7e7cf4f24ce3bed276c1860aa041 with delete_type TABLET_DATA_TOMBSTONED (TS 52a58bb8fe574f5ab1485c59bf994570 not found in new config with opid_index 302) from {username='slave'} at 127.0.0.1:36310
I20240503 13:59:48.911427 9905 tablet_replica.cc:330] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570: stopping tablet replica
I20240503 13:59:48.911746 9905 raft_consensus.cc:2229] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:48.911927 9905 raft_consensus.cc:2258] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:48.912611 9905 ts_tablet_manager.cc:1875] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:48.914999 9905 ts_tablet_manager.cc:1888] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.301
I20240503 13:59:48.915191 9905 log.cc:1196] T 527c7e7cf4f24ce3bed276c1860aa041 P 52a58bb8fe574f5ab1485c59bf994570: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/527c7e7cf4f24ce3bed276c1860aa041
I20240503 13:59:48.916054 6620 catalog_manager.cc:4830] TS 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439): tablet 527c7e7cf4f24ce3bed276c1860aa041 (table rebalance_test_table_0 [id=f9d13cbf321a4a76b3f59808a6da0969]) successfully deleted
I20240503 13:59:49.162552 7020 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:49.225759 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:49.232687 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:49.233335 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:49.250984 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:49.273645 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:49.279342 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:49.511628 6908 consensus_queue.cc:237] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 394, Committed index: 394, Last appended: 1.395, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:49.512845 8420 raft_consensus.cc:1261] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 395. Preceding OpId from leader: term: 1 index: 396. (index mismatch)
I20240503 13:59:49.513623 9764 consensus_queue.cc:1035] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 394, Time since last communication: 0.000s
I20240503 13:59:49.515563 9764 raft_consensus.cc:2932] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.396: config changed from index 92 to 396, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:49.518028 6633 catalog_manager.cc:5484] T 4f882dbedd134f2998642ac747567db0 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: config changed from index 92 to 396, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:49.518077 8420 raft_consensus.cc:2932] T 4f882dbedd134f2998642ac747567db0 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Committing config change with OpId 1.396: config changed from index 92 to 396, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:49.556202 6758 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f882dbedd134f2998642ac747567db0 with delete_type TABLET_DATA_TOMBSTONED (TS 791cddb67b7e415aad7edb5151037418 not found in new config with opid_index 396) from {username='slave'} at 127.0.0.1:40392
I20240503 13:59:49.559286 9969 tablet_replica.cc:330] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418: stopping tablet replica
I20240503 13:59:49.559520 9969 raft_consensus.cc:2229] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:49.559684 9969 pending_rounds.cc:62] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418: Trying to abort 1 pending ops.
I20240503 13:59:49.559769 9969 pending_rounds.cc:68] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418: Aborting op as it isn't in flight: id { term: 1 index: 393 } timestamp: 7023594657776246784 op_type: WRITE_OP write_request { tablet_id: "4f882dbedd134f2998642ac747567db0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\255\277Yb)wx\003\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5e112abf721ae209" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023594657760636928 authz_token { token_data: "\010\365\335\323\261\006\"3\n\005slave\022*\n bffd930039c544a68647ac9a405cf590\020\001\030\001 \001(\001" signature: "<redacted>""\305\233C%\221\241f\233\364[\014\262F3!\025Z\344o\344\251S\355\376\177\227\236%=zdR\304\020\222\035>\277\376\363\236_?\210I\304\003L\305\300\032_s\344\256C\330\346\222J\372 =\020" signing_key_seq_num: 0 } } request_id { client_id: "d3d84140f9384e439b2d83449f2e5f65" seq_no: 1214 first_incomplete_seq_no: 1214 attempt_no: 0 }
I20240503 13:59:49.560148 9969 raft_consensus.cc:2258] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:49.562064 9969 ts_tablet_manager.cc:1875] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:49.565985 9969 ts_tablet_manager.cc:1888] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.393
I20240503 13:59:49.566154 9969 log.cc:1196] T 4f882dbedd134f2998642ac747567db0 P 791cddb67b7e415aad7edb5151037418: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4f882dbedd134f2998642ac747567db0
I20240503 13:59:49.566840 6620 catalog_manager.cc:4830] TS 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893): tablet 4f882dbedd134f2998642ac747567db0 (table rebalance_test_table_3 [id=bffd930039c544a68647ac9a405cf590]) successfully deleted
I20240503 13:59:49.797750 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:49.819423 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:49.827790 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:49.829551 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:49.843389 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:49.864662 7020 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:49.896934 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:50.136286 7039 consensus_queue.cc:237] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 377, Committed index: 377, Last appended: 1.377, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:50.138041 8420 raft_consensus.cc:1261] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 377. Preceding OpId from leader: term: 1 index: 378. (index mismatch)
I20240503 13:59:50.138464 9947 consensus_queue.cc:1035] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 378, Last known committed idx: 377, Time since last communication: 0.000s
I20240503 13:59:50.140229 8557 raft_consensus.cc:2932] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Committing config change with OpId 1.378: config changed from index 27 to 378, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:50.141043 8420 raft_consensus.cc:2932] T 9a52cb95109045d2bbfa3def26848224 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Committing config change with OpId 1.378: config changed from index 27 to 378, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:50.144210 6633 catalog_manager.cc:5484] T 9a52cb95109045d2bbfa3def26848224 P 14c195eab0b34945afd10f3291ed080b reported cstate change: config changed from index 27 to 378, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:50.265501 6758 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a52cb95109045d2bbfa3def26848224 with delete_type TABLET_DATA_TOMBSTONED (TS 791cddb67b7e415aad7edb5151037418 not found in new config with opid_index 378) from {username='slave'} at 127.0.0.1:40392
I20240503 13:59:50.266777 10057 tablet_replica.cc:330] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418: stopping tablet replica
I20240503 13:59:50.266960 10057 raft_consensus.cc:2229] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:50.267133 10057 raft_consensus.cc:2258] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:50.268725 10057 ts_tablet_manager.cc:1875] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:50.273346 10057 ts_tablet_manager.cc:1888] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.377
I20240503 13:59:50.273572 10057 log.cc:1196] T 9a52cb95109045d2bbfa3def26848224 P 791cddb67b7e415aad7edb5151037418: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9a52cb95109045d2bbfa3def26848224
I20240503 13:59:50.277578 6620 catalog_manager.cc:4830] TS 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893): tablet 9a52cb95109045d2bbfa3def26848224 (table rebalance_test_table_2 [id=04d1a2edbd1a48bc84eae4ce2c7379ff]) successfully deleted
I20240503 13:59:50.340572 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:50.342118 7020 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:50.383749 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:50.395992 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:50.414376 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:50.423790 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:50.426220 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:50.577463 7039 consensus_queue.cc:237] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 376, Committed index: 376, Last appended: 1.376, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } }
I20240503 13:59:50.578595 8022 raft_consensus.cc:1261] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 376. Preceding OpId from leader: term: 1 index: 377. (index mismatch)
I20240503 13:59:50.580358 7819 consensus_queue.cc:1035] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 377, Last known committed idx: 376, Time since last communication: 0.000s
I20240503 13:59:50.591152 8557 raft_consensus.cc:2932] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Committing config change with OpId 1.377: config changed from index 41 to 377, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:50.595793 6631 catalog_manager.cc:5484] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b reported cstate change: config changed from index 41 to 377, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:50.596691 8022 raft_consensus.cc:2932] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Committing config change with OpId 1.377: config changed from index 41 to 377, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } }
I20240503 13:59:50.603040 6889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aec442fecd64614858bce4a5bce5db4 with delete_type TABLET_DATA_TOMBSTONED (TS 52a58bb8fe574f5ab1485c59bf994570 not found in new config with opid_index 377) from {username='slave'} at 127.0.0.1:36310
I20240503 13:59:50.603695 10103 tablet_replica.cc:330] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570: stopping tablet replica
I20240503 13:59:50.603868 10103 raft_consensus.cc:2229] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:50.604051 10103 raft_consensus.cc:2258] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:50.604939 10103 ts_tablet_manager.cc:1875] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:50.617758 10103 ts_tablet_manager.cc:1888] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.376
I20240503 13:59:50.617905 10103 log.cc:1196] T 4aec442fecd64614858bce4a5bce5db4 P 52a58bb8fe574f5ab1485c59bf994570: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4aec442fecd64614858bce4a5bce5db4
I20240503 13:59:50.619093 6620 catalog_manager.cc:4830] TS 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439): tablet 4aec442fecd64614858bce4a5bce5db4 (table rebalance_test_table_4 [id=1b4f4315b2d34ad8a7a1bdd75ec294ef]) successfully deleted
I20240503 13:59:50.803997 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:50.807147 7020 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:50.812361 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:50.814901 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:50.831141 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:50.848340 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:50.900418 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:50.996156 7039 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d118ea0cbd824762906095329740d0a3"
dest_uuid: "14c195eab0b34945afd10f3291ed080b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54110
I20240503 13:59:50.996306 7039 raft_consensus.cc:595] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Received request to transfer leadership
W20240503 13:59:51.021095 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.025797 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.034384 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.043323 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.053800 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.067487 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.075951 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.087867 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.101740 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.116160 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.130512 6998 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.144577 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.162092 6998 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.181216 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.197829 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.219079 6998 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.238369 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.257455 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.279898 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
I20240503 13:59:51.284453 7020 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 13:59:51.304744 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 13:59:51.306164 6999 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
W20240503 13:59:51.327817 6998 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
I20240503 13:59:51.348826 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 13:59:51.352115 6998 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Service unavailable: leader transfer in progress
I20240503 13:59:51.361359 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:51.363858 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:51.363858 8401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:51.364908 8557 raft_consensus.cc:979] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: : Instructing follower 4f82135907754dd8b69d08d6b7eae847 to start an election
I20240503 13:59:51.365002 8557 raft_consensus.cc:1067] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Signalling peer 4f82135907754dd8b69d08d6b7eae847 to start an election
I20240503 13:59:51.367595 8420 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d118ea0cbd824762906095329740d0a3"
dest_uuid: "4f82135907754dd8b69d08d6b7eae847"
from {username='slave'} at 127.0.99.131:43163
I20240503 13:59:51.367791 8420 raft_consensus.cc:482] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:51.367882 8420 raft_consensus.cc:3037] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:51.368906 8420 raft_consensus.cc:504] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:51.371940 8420 leader_election.cc:288] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [CANDIDATE]: Term 2 election: Requested vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699), 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893)
I20240503 13:59:51.372406 7039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d118ea0cbd824762906095329740d0a3" candidate_uuid: "4f82135907754dd8b69d08d6b7eae847" candidate_term: 2 candidate_status { last_received { term: 1 index: 403 } } ignore_live_leader: true dest_uuid: "14c195eab0b34945afd10f3291ed080b"
I20240503 13:59:51.372505 7039 raft_consensus.cc:3032] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:51.372536 7039 raft_consensus.cc:729] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Running, Role: LEADER
I20240503 13:59:51.372644 7039 consensus_queue.cc:260] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 403, Committed index: 403, Last appended: 1.403, Last appended by leader: 403, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:51.372752 7039 raft_consensus.cc:3037] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:51.373713 7039 raft_consensus.cc:2454] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f82135907754dd8b69d08d6b7eae847 in term 2.
I20240503 13:59:51.374634 8354 leader_election.cc:302] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 4f82135907754dd8b69d08d6b7eae847; no voters:
I20240503 13:59:51.376219 6776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d118ea0cbd824762906095329740d0a3" candidate_uuid: "4f82135907754dd8b69d08d6b7eae847" candidate_term: 2 candidate_status { last_received { term: 1 index: 403 } } ignore_live_leader: true dest_uuid: "791cddb67b7e415aad7edb5151037418"
I20240503 13:59:51.376324 6776 raft_consensus.cc:3037] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:51.377112 6776 raft_consensus.cc:2454] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f82135907754dd8b69d08d6b7eae847 in term 2.
W20240503 13:59:51.381726 6998 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42420: Illegal state: replica 14c195eab0b34945afd10f3291ed080b is not leader of this config: current role FOLLOWER
I20240503 13:59:51.385066 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:51.397600 10193 raft_consensus.cc:2781] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:51.397898 10193 raft_consensus.cc:686] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [term 2 LEADER]: Becoming Leader. State: Replica: 4f82135907754dd8b69d08d6b7eae847, State: Running, Role: LEADER
I20240503 13:59:51.398274 10193 consensus_queue.cc:237] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 403, Committed index: 403, Last appended: 1.403, Last appended by leader: 403, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:51.404256 6633 catalog_manager.cc:5484] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 reported cstate change: term changed from 1 to 2, leader changed from 14c195eab0b34945afd10f3291ed080b (127.0.99.131) to 4f82135907754dd8b69d08d6b7eae847 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "4f82135907754dd8b69d08d6b7eae847" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
W20240503 13:59:51.449240 6737 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41232: Illegal state: replica 791cddb67b7e415aad7edb5151037418 is not leader of this config: current role FOLLOWER
I20240503 13:59:51.486057 6775 raft_consensus.cc:1261] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Refusing update from remote peer 4f82135907754dd8b69d08d6b7eae847: Log matching property violated. Preceding OpId in replica: term: 1 index: 403. Preceding OpId from leader: term: 2 index: 405. (index mismatch)
I20240503 13:59:51.486497 7038 raft_consensus.cc:1261] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Refusing update from remote peer 4f82135907754dd8b69d08d6b7eae847: Log matching property violated. Preceding OpId in replica: term: 1 index: 403. Preceding OpId from leader: term: 2 index: 405. (index mismatch)
I20240503 13:59:51.486486 10193 consensus_queue.cc:1035] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 404, Last known committed idx: 403, Time since last communication: 0.000s
I20240503 13:59:51.487039 10193 consensus_queue.cc:1035] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 404, Last known committed idx: 403, Time since last communication: 0.000s
I20240503 13:59:51.489905 9667 mvcc.cc:205] Tried to move back new op lower bound from 7023594665921892352 to 7023594665578610688. Current Snapshot: MvccSnapshot[applied={T|T < 7023594665921892352 or (T in {7023594665921892352})}]
I20240503 13:59:51.491809 10167 mvcc.cc:205] Tried to move back new op lower bound from 7023594665921892352 to 7023594665578610688. Current Snapshot: MvccSnapshot[applied={T|T < 7023594665921892352}]
I20240503 13:59:51.636281 8419 consensus_queue.cc:237] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 408, Committed index: 408, Last appended: 2.409, Last appended by leader: 403, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } }
I20240503 13:59:51.637647 6775 raft_consensus.cc:1261] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Refusing update from remote peer 4f82135907754dd8b69d08d6b7eae847: Log matching property violated. Preceding OpId in replica: term: 2 index: 409. Preceding OpId from leader: term: 2 index: 410. (index mismatch)
I20240503 13:59:51.638424 10194 consensus_queue.cc:1035] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 410, Last known committed idx: 408, Time since last communication: 0.000s
I20240503 13:59:51.640352 10194 raft_consensus.cc:2932] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 [term 2 LEADER]: Committing config change with OpId 2.410: config changed from index 39 to 410, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:51.644001 6633 catalog_manager.cc:5484] T d118ea0cbd824762906095329740d0a3 P 4f82135907754dd8b69d08d6b7eae847 reported cstate change: config changed from index 39 to 410, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "4f82135907754dd8b69d08d6b7eae847" committed_config { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:51.645408 6775 raft_consensus.cc:2932] T d118ea0cbd824762906095329740d0a3 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Committing config change with OpId 2.410: config changed from index 39 to 410, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4f82135907754dd8b69d08d6b7eae847" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 34409 } } }
I20240503 13:59:51.648757 7020 tablet_service.cc:1514] Processing DeleteTablet for tablet d118ea0cbd824762906095329740d0a3 with delete_type TABLET_DATA_TOMBSTONED (TS 14c195eab0b34945afd10f3291ed080b not found in new config with opid_index 410) from {username='slave'} at 127.0.0.1:42338
I20240503 13:59:51.649220 10221 tablet_replica.cc:330] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: stopping tablet replica
I20240503 13:59:51.649390 10221 raft_consensus.cc:2229] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:51.649485 10221 pending_rounds.cc:62] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: Trying to abort 1 pending ops.
I20240503 13:59:51.649550 10221 pending_rounds.cc:68] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: Aborting op as it isn't in flight: id { term: 2 index: 409 } timestamp: 7023594666533728256 op_type: WRITE_OP write_request { tablet_id: "d118ea0cbd824762906095329740d0a3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\2144%7\366!7\014\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a16c0550f5a1f24f" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023594666480578560 authz_token { token_data: "\010\365\335\323\261\006\"3\n\005slave\022*\n 1b4f4315b2d34ad8a7a1bdd75ec294ef\020\001\030\001 \001(\001" signature: "<redacted>""\035\342U\211\374\326\225\360Jv@y\201\2134\361\367v\327:\2641{\004(\241E\n\267\030\031\250\342\325S{\3272^\205\034V3R\245\035\363[^\345\0260\022\007_9\300\010\003i\266\347kK" signing_key_seq_num: 0 } } request_id { client_id: "12983e91c3d648edbbef278864ab2d9e" seq_no: 1219 first_incomplete_seq_no: 1219 attempt_no: 0 }
I20240503 13:59:51.651178 10221 raft_consensus.cc:2258] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:51.652410 10221 ts_tablet_manager.cc:1875] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:51.658955 10221 ts_tablet_manager.cc:1888] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.409
I20240503 13:59:51.659116 10221 log.cc:1196] T d118ea0cbd824762906095329740d0a3 P 14c195eab0b34945afd10f3291ed080b: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d118ea0cbd824762906095329740d0a3
I20240503 13:59:51.660060 6620 catalog_manager.cc:4830] TS 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699): tablet d118ea0cbd824762906095329740d0a3 (table rebalance_test_table_4 [id=1b4f4315b2d34ad8a7a1bdd75ec294ef]) successfully deleted
I20240503 13:59:51.883925 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:51.919075 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:51.958863 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:51.967299 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:51.969332 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:51.970131 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:51.970376 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:52.103018 6908 consensus_queue.cc:237] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 471, Committed index: 471, Last appended: 1.471, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:52.104403 8286 raft_consensus.cc:1261] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 471. Preceding OpId from leader: term: 1 index: 472. (index mismatch)
I20240503 13:59:52.104861 10266 consensus_queue.cc:1035] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 472, Last known committed idx: 471, Time since last communication: 0.000s
I20240503 13:59:52.107133 8548 raft_consensus.cc:2932] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.472: config changed from index 61 to 472, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:52.109001 8285 raft_consensus.cc:2932] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Committing config change with OpId 1.472: config changed from index 61 to 472, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:52.111313 6633 catalog_manager.cc:5484] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f reported cstate change: config changed from index 61 to 472, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 472 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:52.114051 6758 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a9724b166dc4ca690fc912049779985 with delete_type TABLET_DATA_TOMBSTONED (TS 791cddb67b7e415aad7edb5151037418 not found in new config with opid_index 472) from {username='slave'} at 127.0.0.1:40392
I20240503 13:59:52.114594 10284 tablet_replica.cc:330] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418: stopping tablet replica
I20240503 13:59:52.114756 10284 raft_consensus.cc:2229] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.114869 10284 raft_consensus.cc:2258] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.115761 10284 ts_tablet_manager.cc:1875] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.118650 10284 ts_tablet_manager.cc:1888] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.471
I20240503 13:59:52.118760 10284 log.cc:1196] T 4a9724b166dc4ca690fc912049779985 P 791cddb67b7e415aad7edb5151037418: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4a9724b166dc4ca690fc912049779985
I20240503 13:59:52.119411 6620 catalog_manager.cc:4830] TS 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893): tablet 4a9724b166dc4ca690fc912049779985 (table rebalance_test_table_1 [id=bb88829318084b1e8ec739465823de3c]) successfully deleted
I20240503 13:59:52.507175 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:52.516831 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:52.530071 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:52.543399 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:52.575933 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:52.613096 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:52.643942 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:52.724074 6908 consensus_queue.cc:237] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 474, Committed index: 474, Last appended: 1.474, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } }
I20240503 13:59:52.725409 8286 raft_consensus.cc:1261] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 473. Preceding OpId from leader: term: 1 index: 475. (index mismatch)
I20240503 13:59:52.725898 9582 consensus_queue.cc:1035] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 475, Last known committed idx: 473, Time since last communication: 0.000s
I20240503 13:59:52.726819 8286 raft_consensus.cc:1203] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.473->[1.474-1.476] Dedup: 1.474->[1.475-1.476]
I20240503 13:59:52.736454 9199 raft_consensus.cc:2932] T ef2d39830da24221a3dc44da57c0a6e8 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.475: config changed from index 89 to 475, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:52.739318 8286 raft_consensus.cc:2932] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Committing config change with OpId 1.475: config changed from index 89 to 475, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:52.742445 6633 catalog_manager.cc:5484] T ef2d39830da24221a3dc44da57c0a6e8 P 083b1c5c79a04af899eea33ba9f4502f reported cstate change: config changed from index 89 to 475, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } }
I20240503 13:59:52.786785 7020 tablet_service.cc:1514] Processing DeleteTablet for tablet ef2d39830da24221a3dc44da57c0a6e8 with delete_type TABLET_DATA_TOMBSTONED (TS 14c195eab0b34945afd10f3291ed080b not found in new config with opid_index 475) from {username='slave'} at 127.0.0.1:42338
I20240503 13:59:52.790472 10360 tablet_replica.cc:330] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b: stopping tablet replica
I20240503 13:59:52.790670 10360 raft_consensus.cc:2229] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:52.791759 10360 raft_consensus.cc:2258] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:52.792466 10360 ts_tablet_manager.cc:1875] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:52.800240 10360 ts_tablet_manager.cc:1888] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.474
I20240503 13:59:52.800359 10360 log.cc:1196] T ef2d39830da24221a3dc44da57c0a6e8 P 14c195eab0b34945afd10f3291ed080b: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ef2d39830da24221a3dc44da57c0a6e8
I20240503 13:59:52.801596 6620 catalog_manager.cc:4830] TS 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699): tablet ef2d39830da24221a3dc44da57c0a6e8 (table rebalance_test_table_2 [id=04d1a2edbd1a48bc84eae4ce2c7379ff]) successfully deleted
I20240503 13:59:53.603668 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:53.634775 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:53.637310 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:53.637318 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:53.663959 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:53.666471 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:53.671020 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:53.914119 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:53.915580 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:53.915943 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:53.917678 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:53.930164 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:53.930864 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:53.932101 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:54.609061 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:54.615383 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:54.628671 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:54.673288 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:54.707453 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:54.748258 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:54.759114 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:55.032747 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:55.042671 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:55.043087 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:55.056449 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:55.060156 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:55.088739 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:55.098814 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:55.289992 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:55.291769 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:55.293315 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:55.299438 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:55.302029 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:55.311833 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:55.312407 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
11a95c76ed294a80a8da622eab79df3a | 127.0.99.190:42439 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:45383 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
083b1c5c79a04af899eea33ba9f4502f | 127.0.99.134:44579 | HEALTHY | <none> | 1 | 0
14c195eab0b34945afd10f3291ed080b | 127.0.99.131:40699 | HEALTHY | <none> | 5 | 0
40e8eacd02b44bb8bbec0a1dc205a671 | 127.0.99.132:41965 | HEALTHY | <none> | 0 | 0
4c5b7c28dd2a4920a2b4e97d1afe2384 | 127.0.99.133:39967 | HEALTHY | <none> | 1 | 0
4f82135907754dd8b69d08d6b7eae847 | 127.0.99.135:34409 | HEALTHY | <none> | 1 | 0
52a58bb8fe574f5ab1485c59bf994570 | 127.0.99.130:33439 | HEALTHY | <none> | 5 | 0
791cddb67b7e415aad7edb5151037418 | 127.0.99.129:43893 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:43893
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:33439
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:40699
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:41965
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:39967
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:44579
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:34409
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:43893
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:33439
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:40699
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:41965
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:39967
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:44579
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:34409
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:45383 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 13:59:55.347385 398 log_verifier.cc:126] Checking tablet 0a871746624e442597588274959d5e24
I20240503 13:59:55.433980 398 log_verifier.cc:177] Verified matching terms for 537 ops in tablet 0a871746624e442597588274959d5e24
I20240503 13:59:55.434173 398 log_verifier.cc:126] Checking tablet 4a9724b166dc4ca690fc912049779985
I20240503 13:59:55.522475 398 log_verifier.cc:177] Verified matching terms for 570 ops in tablet 4a9724b166dc4ca690fc912049779985
I20240503 13:59:55.522614 398 log_verifier.cc:126] Checking tablet 4aec442fecd64614858bce4a5bce5db4
I20240503 13:59:55.609123 398 log_verifier.cc:177] Verified matching terms for 553 ops in tablet 4aec442fecd64614858bce4a5bce5db4
I20240503 13:59:55.609280 398 log_verifier.cc:126] Checking tablet 4f882dbedd134f2998642ac747567db0
I20240503 13:59:55.700914 398 log_verifier.cc:177] Verified matching terms for 604 ops in tablet 4f882dbedd134f2998642ac747567db0
I20240503 13:59:55.701131 398 log_verifier.cc:126] Checking tablet 527c7e7cf4f24ce3bed276c1860aa041
I20240503 13:59:55.785023 398 log_verifier.cc:177] Verified matching terms for 514 ops in tablet 527c7e7cf4f24ce3bed276c1860aa041
I20240503 13:59:55.785205 398 log_verifier.cc:126] Checking tablet 57ea09e3f1f74d74839ddb0afd90acc2
I20240503 13:59:55.879689 398 log_verifier.cc:177] Verified matching terms for 620 ops in tablet 57ea09e3f1f74d74839ddb0afd90acc2
I20240503 13:59:55.879864 398 log_verifier.cc:126] Checking tablet 5fb5bbc56575455eb122d1e401fbdc64
I20240503 13:59:55.962549 398 log_verifier.cc:177] Verified matching terms for 575 ops in tablet 5fb5bbc56575455eb122d1e401fbdc64
I20240503 13:59:55.962724 398 log_verifier.cc:126] Checking tablet 6409f411c5d24e97b34aa9cce7c5a5ea
I20240503 13:59:56.042173 398 log_verifier.cc:177] Verified matching terms for 504 ops in tablet 6409f411c5d24e97b34aa9cce7c5a5ea
I20240503 13:59:56.042364 398 log_verifier.cc:126] Checking tablet 70583380c4e24b4694fc5477d880d944
I20240503 13:59:56.133503 398 log_verifier.cc:177] Verified matching terms for 572 ops in tablet 70583380c4e24b4694fc5477d880d944
I20240503 13:59:56.133664 398 log_verifier.cc:126] Checking tablet 80d845691ee64bf3a5d6c2df59842a54
I20240503 13:59:56.149075 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 80d845691ee64bf3a5d6c2df59842a54
I20240503 13:59:56.149137 398 log_verifier.cc:126] Checking tablet 9a52cb95109045d2bbfa3def26848224
I20240503 13:59:56.238972 398 log_verifier.cc:177] Verified matching terms for 560 ops in tablet 9a52cb95109045d2bbfa3def26848224
I20240503 13:59:56.239151 398 log_verifier.cc:126] Checking tablet a82b94a4baa84d778cc280d58c18af5b
I20240503 13:59:56.328877 398 log_verifier.cc:177] Verified matching terms for 578 ops in tablet a82b94a4baa84d778cc280d58c18af5b
I20240503 13:59:56.329047 398 log_verifier.cc:126] Checking tablet d118ea0cbd824762906095329740d0a3
I20240503 13:59:56.412493 398 log_verifier.cc:177] Verified matching terms for 535 ops in tablet d118ea0cbd824762906095329740d0a3
I20240503 13:59:56.412683 398 log_verifier.cc:126] Checking tablet e026fe8a768941ce84378f23b3dabff2
I20240503 13:59:56.512245 398 log_verifier.cc:177] Verified matching terms for 648 ops in tablet e026fe8a768941ce84378f23b3dabff2
I20240503 13:59:56.512442 398 log_verifier.cc:126] Checking tablet ef2d39830da24221a3dc44da57c0a6e8
I20240503 13:59:56.603053 398 log_verifier.cc:177] Verified matching terms for 578 ops in tablet ef2d39830da24221a3dc44da57c0a6e8
I20240503 13:59:56.603246 398 log_verifier.cc:126] Checking tablet f2b880960ed44b20b81ddf91c8534638
I20240503 13:59:56.693512 398 log_verifier.cc:177] Verified matching terms for 605 ops in tablet f2b880960ed44b20b81ddf91c8534638
I20240503 13:59:56.695418 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--builtin_ntp_servers=127.0.99.148:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 13:59:56.799016 10640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 13:59:56.799175 10640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 13:59:56.799208 10640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 13:59:56.801689 10640 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 13:59:56.801738 10640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 13:59:56.801790 10640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 13:59:56.804690 10640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42439
--never_fsync=true
--heap_profile_path=/tmp/kudu.10640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 13:59:56.805346 10640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:56.806380 10640 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:56.814098 10646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.815157 10649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:56.815917 10647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:56.816612 10640 server_base.cc:1034] running on GCE node
I20240503 13:59:56.816885 10640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 13:59:56.817297 10640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 13:59:56.818464 10640 hybrid_clock.cc:648] HybridClock initialized: now 1714744796818451 us; error 21 us; skew 500 ppm
I20240503 13:59:56.820675 10640 webserver.cc:460] Webserver started at http://127.0.99.136:45517/ using document root <none> and password file <none>
I20240503 13:59:56.821206 10640 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:56.821265 10640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:56.821419 10640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:56.822767 10640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "978d8b072e9d4bd49e227470e16ccfb1"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-snql"
I20240503 13:59:56.823160 10640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "978d8b072e9d4bd49e227470e16ccfb1"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-snql"
I20240503 13:59:56.825881 10640 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:56.827865 10655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.828737 10640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:56.828836 10640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "978d8b072e9d4bd49e227470e16ccfb1"
format_stamp: "Formatted at 2024-05-03 13:59:56 on dist-test-slave-snql"
I20240503 13:59:56.828902 10640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:56.838827 10640 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:56.839391 10640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:56.839525 10640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:56.839946 10640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:56.840731 10640 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:56.840783 10640 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.840831 10640 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:56.840858 10640 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:56.848841 10640 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:32963
I20240503 13:59:56.848877 10768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:32963 every 8 connection(s)
I20240503 13:59:56.849581 10640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 13:59:56.850927 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 10640
I20240503 13:59:56.851042 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 13:59:56.858299 10769 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42439
I20240503 13:59:56.858515 10769 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:56.859006 10769 heartbeater.cc:511] Master 127.0.99.190:42439 requested a full tablet report, sending...
I20240503 13:59:56.859896 6636 ts_manager.cc:194] Registered new tserver with Master: 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136:32963)
I20240503 13:59:56.860592 6636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:39037
I20240503 13:59:57.030213 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.032544 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.042115 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.043859 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.045496 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.046015 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:57.052565 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.059810 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.146288 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.148301 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.148290 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.154232 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:57.154816 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.155964 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.164280 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.185288 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.220160 8285 consensus_queue.cc:237] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:57.221309 8021 raft_consensus.cc:1261] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 2 FOLLOWER]: Refusing update from remote peer 083b1c5c79a04af899eea33ba9f4502f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 13:59:57.221643 10876 consensus_queue.cc:1035] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 13:59:57.223134 10638 raft_consensus.cc:2932] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:57.224284 8021 raft_consensus.cc:2932] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
W20240503 13:59:57.225003 8220 consensus_peers.cc:485] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f -> Peer 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136:32963): Couldn't send request to peer 978d8b072e9d4bd49e227470e16ccfb1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80d845691ee64bf3a5d6c2df59842a54. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:57.225855 6634 catalog_manager.cc:5484] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f reported cstate change: config changed from index -1 to 3, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "083b1c5c79a04af899eea33ba9f4502f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:57.241674 7038 consensus_queue.cc:237] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 553, Committed index: 553, Last appended: 1.553, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 554 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:57.242765 8021 raft_consensus.cc:1261] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Refusing update from remote peer 14c195eab0b34945afd10f3291ed080b: Log matching property violated. Preceding OpId in replica: term: 1 index: 553. Preceding OpId from leader: term: 1 index: 554. (index mismatch)
I20240503 13:59:57.243073 10821 consensus_queue.cc:1035] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 554, Last known committed idx: 553, Time since last communication: 0.000s
I20240503 13:59:57.244488 8557 raft_consensus.cc:2932] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Committing config change with OpId 1.554: config changed from index 377 to 554, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New config: { opid_index: 554 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:57.244906 8021 raft_consensus.cc:2932] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Committing config change with OpId 1.554: config changed from index 377 to 554, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New config: { opid_index: 554 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
W20240503 13:59:57.245821 6973 consensus_peers.cc:485] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b -> Peer 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136:32963): Couldn't send request to peer 978d8b072e9d4bd49e227470e16ccfb1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aec442fecd64614858bce4a5bce5db4. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:57.247463 6631 catalog_manager.cc:5484] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b reported cstate change: config changed from index 377 to 554, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "14c195eab0b34945afd10f3291ed080b" committed_config { opid_index: 554 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:57.261080 6908 consensus_queue.cc:237] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 570, Committed index: 570, Last appended: 1.570, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:57.262444 8285 raft_consensus.cc:1261] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Refusing update from remote peer 52a58bb8fe574f5ab1485c59bf994570: Log matching property violated. Preceding OpId in replica: term: 1 index: 570. Preceding OpId from leader: term: 1 index: 571. (index mismatch)
I20240503 13:59:57.262802 10918 consensus_queue.cc:1035] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 571, Last known committed idx: 570, Time since last communication: 0.000s
I20240503 13:59:57.264765 8548 raft_consensus.cc:2932] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Committing config change with OpId 1.571: config changed from index 472 to 571, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New config: { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
W20240503 13:59:57.265134 6842 consensus_peers.cc:485] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 -> Peer 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136:32963): Couldn't send request to peer 978d8b072e9d4bd49e227470e16ccfb1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a9724b166dc4ca690fc912049779985. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:57.265146 8285 raft_consensus.cc:2932] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Committing config change with OpId 1.571: config changed from index 472 to 571, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New config: { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:57.267338 6634 catalog_manager.cc:5484] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 reported cstate change: config changed from index 472 to 571, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "52a58bb8fe574f5ab1485c59bf994570" committed_config { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:57.284235 8154 consensus_queue.cc:237] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 620, Committed index: 620, Last appended: 2.620, Last appended by leader: 353, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 621 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:57.285643 6775 raft_consensus.cc:1261] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Refusing update from remote peer 4c5b7c28dd2a4920a2b4e97d1afe2384: Log matching property violated. Preceding OpId in replica: term: 2 index: 620. Preceding OpId from leader: term: 2 index: 621. (index mismatch)
I20240503 13:59:57.286087 10937 consensus_queue.cc:1035] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 621, Last known committed idx: 620, Time since last communication: 0.000s
I20240503 13:59:57.287796 10641 raft_consensus.cc:2932] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 2 LEADER]: Committing config change with OpId 2.621: config changed from index 371 to 621, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New config: { opid_index: 621 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:57.288232 6775 raft_consensus.cc:2932] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Committing config change with OpId 2.621: config changed from index 371 to 621, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New config: { opid_index: 621 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
W20240503 13:59:57.288388 8089 consensus_peers.cc:485] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 -> Peer 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136:32963): Couldn't send request to peer 978d8b072e9d4bd49e227470e16ccfb1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57ea09e3f1f74d74839ddb0afd90acc2. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:57.289937 6631 catalog_manager.cc:5484] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 reported cstate change: config changed from index 371 to 621, VOTER 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" committed_config { opid_index: 621 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:57.344081 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.344339 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.344434 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.357534 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.358094 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.359351 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.359719 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.360354 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:57.449487 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.449949 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.450461 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.457628 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.463315 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:57.463641 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.474238 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.481050 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.552886 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.553346 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.559176 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.560601 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.561682 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:57.565866 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.566965 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.580404 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.649492 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.650969 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.655939 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.659404 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.659705 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:57.661331 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.666963 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.678762 11090 ts_tablet_manager.cc:902] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1: Initiating tablet copy from peer 4c5b7c28dd2a4920a2b4e97d1afe2384 (127.0.99.133:39967)
I20240503 13:59:57.679607 11090 tablet_copy_client.cc:308] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:39967
I20240503 13:59:57.679849 11089 ts_tablet_manager.cc:902] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1: Initiating tablet copy from peer 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579)
I20240503 13:59:57.681059 11089 tablet_copy_client.cc:308] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:44579
I20240503 13:59:57.684432 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.686250 8176 tablet_copy_service.cc:140] P 4c5b7c28dd2a4920a2b4e97d1afe2384: Received BeginTabletCopySession request for tablet 57ea09e3f1f74d74839ddb0afd90acc2 from peer 978d8b072e9d4bd49e227470e16ccfb1 ({username='slave'} at 127.0.99.136:46199)
I20240503 13:59:57.686389 8176 tablet_copy_service.cc:161] P 4c5b7c28dd2a4920a2b4e97d1afe2384: Beginning new tablet copy session on tablet 57ea09e3f1f74d74839ddb0afd90acc2 from peer 978d8b072e9d4bd49e227470e16ccfb1 at {username='slave'} at 127.0.99.136:46199: session id = 978d8b072e9d4bd49e227470e16ccfb1-57ea09e3f1f74d74839ddb0afd90acc2
I20240503 13:59:57.687613 8176 tablet_copy_source_session.cc:205] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.689395 8308 tablet_copy_service.cc:140] P 083b1c5c79a04af899eea33ba9f4502f: Received BeginTabletCopySession request for tablet 80d845691ee64bf3a5d6c2df59842a54 from peer 978d8b072e9d4bd49e227470e16ccfb1 ({username='slave'} at 127.0.99.136:58319)
I20240503 13:59:57.689512 8308 tablet_copy_service.cc:161] P 083b1c5c79a04af899eea33ba9f4502f: Beginning new tablet copy session on tablet 80d845691ee64bf3a5d6c2df59842a54 from peer 978d8b072e9d4bd49e227470e16ccfb1 at {username='slave'} at 127.0.99.136:58319: session id = 978d8b072e9d4bd49e227470e16ccfb1-80d845691ee64bf3a5d6c2df59842a54
I20240503 13:59:57.690485 8308 tablet_copy_source_session.cc:205] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.706092 11090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57ea09e3f1f74d74839ddb0afd90acc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.706524 11089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d845691ee64bf3a5d6c2df59842a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.711995 11090 tablet_copy_client.cc:783] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.712178 11090 tablet_copy_client.cc:647] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.712720 11089 tablet_copy_client.cc:783] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.712872 11089 tablet_copy_client.cc:647] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.714267 11089 tablet_copy_client.cc:523] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.715690 11089 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap starting.
I20240503 13:59:57.715740 11090 tablet_copy_client.cc:523] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.717020 11090 tablet_bootstrap.cc:492] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap starting.
I20240503 13:59:57.720865 11089 log.cc:824] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:57.723263 11108 ts_tablet_manager.cc:902] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1: Initiating tablet copy from peer 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439)
I20240503 13:59:57.723562 11089 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:57.723697 11108 tablet_copy_client.cc:308] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33439
I20240503 13:59:57.724011 11089 tablet_bootstrap.cc:492] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap complete.
I20240503 13:59:57.725653 11089 ts_tablet_manager.cc:1367] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240503 13:59:57.729627 11089 raft_consensus.cc:348] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:57.729925 11089 raft_consensus.cc:729] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978d8b072e9d4bd49e227470e16ccfb1, State: Initialized, Role: FOLLOWER
I20240503 13:59:57.730372 6929 tablet_copy_service.cc:140] P 52a58bb8fe574f5ab1485c59bf994570: Received BeginTabletCopySession request for tablet 4a9724b166dc4ca690fc912049779985 from peer 978d8b072e9d4bd49e227470e16ccfb1 ({username='slave'} at 127.0.99.136:46243)
I20240503 13:59:57.730471 6929 tablet_copy_service.cc:161] P 52a58bb8fe574f5ab1485c59bf994570: Beginning new tablet copy session on tablet 4a9724b166dc4ca690fc912049779985 from peer 978d8b072e9d4bd49e227470e16ccfb1 at {username='slave'} at 127.0.99.136:46243: session id = 978d8b072e9d4bd49e227470e16ccfb1-4a9724b166dc4ca690fc912049779985
I20240503 13:59:57.730433 11089 consensus_queue.cc:260] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:57.731181 6929 tablet_copy_source_session.cc:205] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.732095 11108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9724b166dc4ca690fc912049779985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.733318 11089 ts_tablet_manager.cc:1398] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1: Time spent starting tablet: real 0.008s user 0.000s sys 0.005s
I20240503 13:59:57.733673 10769 heartbeater.cc:503] Master 127.0.99.190:42439 was elected leader, sending a full tablet report...
I20240503 13:59:57.733974 8308 tablet_copy_service.cc:342] P 083b1c5c79a04af899eea33ba9f4502f: Request end of tablet copy session 978d8b072e9d4bd49e227470e16ccfb1-80d845691ee64bf3a5d6c2df59842a54 received from {username='slave'} at 127.0.99.136:58319
I20240503 13:59:57.734066 8308 tablet_copy_service.cc:433] P 083b1c5c79a04af899eea33ba9f4502f: ending tablet copy session 978d8b072e9d4bd49e227470e16ccfb1-80d845691ee64bf3a5d6c2df59842a54 on tablet 80d845691ee64bf3a5d6c2df59842a54 with peer 978d8b072e9d4bd49e227470e16ccfb1
I20240503 13:59:57.734372 11108 tablet_copy_client.cc:783] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.734503 11108 tablet_copy_client.cc:647] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.738278 11108 tablet_copy_client.cc:523] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.739573 11108 tablet_bootstrap.cc:492] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap starting.
I20240503 13:59:57.765095 11089 ts_tablet_manager.cc:902] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1: Initiating tablet copy from peer 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699)
I20240503 13:59:57.765534 11089 tablet_copy_client.cc:308] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:40699
I20240503 13:59:57.770015 7060 tablet_copy_service.cc:140] P 14c195eab0b34945afd10f3291ed080b: Received BeginTabletCopySession request for tablet 4aec442fecd64614858bce4a5bce5db4 from peer 978d8b072e9d4bd49e227470e16ccfb1 ({username='slave'} at 127.0.99.136:46725)
I20240503 13:59:57.770118 7060 tablet_copy_service.cc:161] P 14c195eab0b34945afd10f3291ed080b: Beginning new tablet copy session on tablet 4aec442fecd64614858bce4a5bce5db4 from peer 978d8b072e9d4bd49e227470e16ccfb1 at {username='slave'} at 127.0.99.136:46725: session id = 978d8b072e9d4bd49e227470e16ccfb1-4aec442fecd64614858bce4a5bce5db4
I20240503 13:59:57.771027 7060 tablet_copy_source_session.cc:205] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 13:59:57.771973 11089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aec442fecd64614858bce4a5bce5db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.774101 11089 tablet_copy_client.cc:783] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Starting download of 0 data blocks...
I20240503 13:59:57.774237 11089 tablet_copy_client.cc:647] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Starting download of 1 WAL segments...
I20240503 13:59:57.779197 11089 tablet_copy_client.cc:523] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 13:59:57.782080 11089 tablet_bootstrap.cc:492] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap starting.
I20240503 13:59:57.786387 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.795727 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.800137 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:57.833565 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.865676 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:57.867136 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:57.869233 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:57.890369 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.011747 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.012069 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.014822 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.027204 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.052757 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.055817 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:58.062489 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.065191 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.185262 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.185562 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.185968 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.221928 10723 raft_consensus.cc:1203] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240503 13:59:58.228202 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.240818 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.241677 11090 tablet_bootstrap.cc:492] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap replayed 1/1 log segments. Stats: ops{read=621 overwritten=0 applied=621 ignored=0} inserts{seen=1249 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:58.242002 11090 tablet_bootstrap.cc:492] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap complete.
I20240503 13:59:58.242244 11090 ts_tablet_manager.cc:1367] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1: Time spent bootstrapping tablet: real 0.525s user 0.385s sys 0.033s
I20240503 13:59:58.242830 11090 raft_consensus.cc:348] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 621 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:58.242944 11090 raft_consensus.cc:729] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978d8b072e9d4bd49e227470e16ccfb1, State: Initialized, Role: FOLLOWER
I20240503 13:59:58.243088 11090 consensus_queue.cc:260] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 621, Last appended: 2.621, Last appended by leader: 621, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 621 OBSOLETE_local: false peers { permanent_uuid: "791cddb67b7e415aad7edb5151037418" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 43893 } } peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:58.245229 11090 ts_tablet_manager.cc:1398] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:58.246099 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:58.247574 8176 tablet_copy_service.cc:342] P 4c5b7c28dd2a4920a2b4e97d1afe2384: Request end of tablet copy session 978d8b072e9d4bd49e227470e16ccfb1-57ea09e3f1f74d74839ddb0afd90acc2 received from {username='slave'} at 127.0.99.136:46199
I20240503 13:59:58.247673 8176 tablet_copy_service.cc:433] P 4c5b7c28dd2a4920a2b4e97d1afe2384: ending tablet copy session 978d8b072e9d4bd49e227470e16ccfb1-57ea09e3f1f74d74839ddb0afd90acc2 on tablet 57ea09e3f1f74d74839ddb0afd90acc2 with peer 978d8b072e9d4bd49e227470e16ccfb1
I20240503 13:59:58.251376 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.277065 11089 tablet_bootstrap.cc:492] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap replayed 1/1 log segments. Stats: ops{read=554 overwritten=0 applied=554 ignored=0} inserts{seen=1224 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:58.277350 11089 tablet_bootstrap.cc:492] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap complete.
I20240503 13:59:58.277560 11089 ts_tablet_manager.cc:1367] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1: Time spent bootstrapping tablet: real 0.496s user 0.321s sys 0.049s
I20240503 13:59:58.277971 11089 raft_consensus.cc:348] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 554 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:58.278085 11089 raft_consensus.cc:729] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978d8b072e9d4bd49e227470e16ccfb1, State: Initialized, Role: FOLLOWER
I20240503 13:59:58.278208 11089 consensus_queue.cc:260] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 554, Last appended: 1.554, Last appended by leader: 554, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 554 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:58.279116 11089 ts_tablet_manager.cc:1398] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 13:59:58.279642 7060 tablet_copy_service.cc:342] P 14c195eab0b34945afd10f3291ed080b: Request end of tablet copy session 978d8b072e9d4bd49e227470e16ccfb1-4aec442fecd64614858bce4a5bce5db4 received from {username='slave'} at 127.0.99.136:46725
I20240503 13:59:58.279724 7060 tablet_copy_service.cc:433] P 14c195eab0b34945afd10f3291ed080b: ending tablet copy session 978d8b072e9d4bd49e227470e16ccfb1-4aec442fecd64614858bce4a5bce5db4 on tablet 4aec442fecd64614858bce4a5bce5db4 with peer 978d8b072e9d4bd49e227470e16ccfb1
I20240503 13:59:58.285284 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.318004 11108 tablet_bootstrap.cc:492] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap replayed 1/1 log segments. Stats: ops{read=571 overwritten=0 applied=571 ignored=0} inserts{seen=1254 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 13:59:58.318296 11108 tablet_bootstrap.cc:492] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1: Bootstrap complete.
I20240503 13:59:58.318509 11108 ts_tablet_manager.cc:1367] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1: Time spent bootstrapping tablet: real 0.579s user 0.341s sys 0.037s
I20240503 13:59:58.318895 11108 raft_consensus.cc:348] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:58.318997 11108 raft_consensus.cc:729] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978d8b072e9d4bd49e227470e16ccfb1, State: Initialized, Role: FOLLOWER
I20240503 13:59:58.319130 11108 consensus_queue.cc:260] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 571, Last appended: 1.571, Last appended by leader: 571, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:58.319439 11108 ts_tablet_manager.cc:1398] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 13:59:58.320073 6929 tablet_copy_service.cc:342] P 52a58bb8fe574f5ab1485c59bf994570: Request end of tablet copy session 978d8b072e9d4bd49e227470e16ccfb1-4a9724b166dc4ca690fc912049779985 received from {username='slave'} at 127.0.99.136:46243
I20240503 13:59:58.320153 6929 tablet_copy_service.cc:433] P 52a58bb8fe574f5ab1485c59bf994570: ending tablet copy session 978d8b072e9d4bd49e227470e16ccfb1-4a9724b166dc4ca690fc912049779985 on tablet 4a9724b166dc4ca690fc912049779985 with peer 978d8b072e9d4bd49e227470e16ccfb1
I20240503 13:59:58.525961 10723 raft_consensus.cc:1203] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.570->[1.571-1.571] Dedup: 1.571->[]
I20240503 13:59:58.569057 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.569092 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.570676 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.579393 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.581319 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.584652 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.585366 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.597390 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:58.650113 10723 raft_consensus.cc:1203] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.553->[1.554-1.554] Dedup: 1.554->[]
I20240503 13:59:58.665959 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.667462 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.667860 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.668113 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.671348 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.671397 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:58.680629 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.699636 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.717821 6908 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4a9724b166dc4ca690fc912049779985"
dest_uuid: "52a58bb8fe574f5ab1485c59bf994570"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44532
I20240503 13:59:58.717947 6908 raft_consensus.cc:595] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:58.763193 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.767576 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.769095 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.774163 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.780616 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.781680 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:58.782349 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.791985 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.832051 10723 raft_consensus.cc:1203] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.620->[2.621-2.621] Dedup: 2.621->[]
I20240503 13:59:58.874815 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.875667 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.875746 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.879747 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:58.879810 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.894793 6889 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.895264 10703 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.895634 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.928568 8285 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80d845691ee64bf3a5d6c2df59842a54"
dest_uuid: "083b1c5c79a04af899eea33ba9f4502f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44580
I20240503 13:59:58.928718 8285 raft_consensus.cc:595] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 2 LEADER]: Received request to transfer leadership
I20240503 13:59:58.972273 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.974678 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:58.976251 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:58.983263 10918 raft_consensus.cc:979] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: : Instructing follower 978d8b072e9d4bd49e227470e16ccfb1 to start an election
I20240503 13:59:58.983347 10918 raft_consensus.cc:1067] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Signalling peer 978d8b072e9d4bd49e227470e16ccfb1 to start an election
I20240503 13:59:58.983740 10723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a9724b166dc4ca690fc912049779985"
dest_uuid: "978d8b072e9d4bd49e227470e16ccfb1"
from {username='slave'} at 127.0.99.130:38261
I20240503 13:59:58.983927 10723 raft_consensus.cc:482] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:58.984051 10723 raft_consensus.cc:3037] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:58.985103 10723 raft_consensus.cc:504] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:58.986265 10723 leader_election.cc:288] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [CANDIDATE]: Term 2 election: Requested vote from peers 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439), 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579)
I20240503 13:59:58.986514 6908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9724b166dc4ca690fc912049779985" candidate_uuid: "978d8b072e9d4bd49e227470e16ccfb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 571 } } ignore_live_leader: true dest_uuid: "52a58bb8fe574f5ab1485c59bf994570"
I20240503 13:59:58.986631 6908 raft_consensus.cc:3032] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:58.986670 6908 raft_consensus.cc:729] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52a58bb8fe574f5ab1485c59bf994570, State: Running, Role: LEADER
I20240503 13:59:58.986640 8285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a9724b166dc4ca690fc912049779985" candidate_uuid: "978d8b072e9d4bd49e227470e16ccfb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 571 } } ignore_live_leader: true dest_uuid: "083b1c5c79a04af899eea33ba9f4502f"
I20240503 13:59:58.986760 8285 raft_consensus.cc:3037] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:58.986789 6908 consensus_queue.cc:260] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 571, Committed index: 571, Last appended: 1.571, Last appended by leader: 571, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:58.986912 6908 raft_consensus.cc:3037] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:58.987828 8285 raft_consensus.cc:2454] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 978d8b072e9d4bd49e227470e16ccfb1 in term 2.
I20240503 13:59:58.987980 6908 raft_consensus.cc:2454] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 978d8b072e9d4bd49e227470e16ccfb1 in term 2.
I20240503 13:59:58.988226 10656 leader_election.cc:302] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083b1c5c79a04af899eea33ba9f4502f, 978d8b072e9d4bd49e227470e16ccfb1; no voters:
I20240503 13:59:58.988288 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:58.988632 11117 raft_consensus.cc:2781] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:58.988791 11117 raft_consensus.cc:686] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 LEADER]: Becoming Leader. State: Replica: 978d8b072e9d4bd49e227470e16ccfb1, State: Running, Role: LEADER
I20240503 13:59:58.989076 11117 consensus_queue.cc:237] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 571, Committed index: 571, Last appended: 1.571, Last appended by leader: 571, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:58.991487 6632 catalog_manager.cc:5484] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 reported cstate change: term changed from 1 to 2, leader changed from 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) to 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "978d8b072e9d4bd49e227470e16ccfb1" committed_config { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 } } peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.000957 6889 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.002921 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.014048 10703 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.014308 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:59.091094 7020 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 13:59:59.091542 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.092360 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:59.095089 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.096715 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.097991 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:59.119213 10703 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.125730 6889 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.152266 7038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4aec442fecd64614858bce4a5bce5db4"
dest_uuid: "14c195eab0b34945afd10f3291ed080b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54414
I20240503 13:59:59.152383 7038 raft_consensus.cc:595] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Received request to transfer leadership
I20240503 13:59:59.184173 10821 raft_consensus.cc:979] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: : Instructing follower 978d8b072e9d4bd49e227470e16ccfb1 to start an election
I20240503 13:59:59.184293 10821 raft_consensus.cc:1067] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Signalling peer 978d8b072e9d4bd49e227470e16ccfb1 to start an election
I20240503 13:59:59.184700 10723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4aec442fecd64614858bce4a5bce5db4"
dest_uuid: "978d8b072e9d4bd49e227470e16ccfb1"
from {username='slave'} at 127.0.99.131:42717
I20240503 13:59:59.184835 10723 raft_consensus.cc:482] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:59.184906 10723 raft_consensus.cc:3037] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:59.185829 10723 raft_consensus.cc:504] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 554 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:59.186364 10723 leader_election.cc:288] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [CANDIDATE]: Term 2 election: Requested vote from peers 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699), 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132:41965)
I20240503 13:59:59.186910 7038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aec442fecd64614858bce4a5bce5db4" candidate_uuid: "978d8b072e9d4bd49e227470e16ccfb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 554 } } ignore_live_leader: true dest_uuid: "14c195eab0b34945afd10f3291ed080b"
I20240503 13:59:59.187001 7038 raft_consensus.cc:3032] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 13:59:59.187026 7038 raft_consensus.cc:729] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 14c195eab0b34945afd10f3291ed080b, State: Running, Role: LEADER
I20240503 13:59:59.187156 7038 consensus_queue.cc:260] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 554, Committed index: 554, Last appended: 1.554, Last appended by leader: 554, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 554 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:59.187361 7038 raft_consensus.cc:3037] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:59.188472 7038 raft_consensus.cc:2454] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 978d8b072e9d4bd49e227470e16ccfb1 in term 2.
I20240503 13:59:59.189612 10656 leader_election.cc:302] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c195eab0b34945afd10f3291ed080b, 978d8b072e9d4bd49e227470e16ccfb1; no voters:
I20240503 13:59:59.189767 11117 raft_consensus.cc:2781] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:59.189863 11117 raft_consensus.cc:686] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 LEADER]: Becoming Leader. State: Replica: 978d8b072e9d4bd49e227470e16ccfb1, State: Running, Role: LEADER
I20240503 13:59:59.189989 11117 consensus_queue.cc:237] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 554, Committed index: 554, Last appended: 1.554, Last appended by leader: 554, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 554 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:59.191835 6633 catalog_manager.cc:5484] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 reported cstate change: term changed from 1 to 2, leader changed from 14c195eab0b34945afd10f3291ed080b (127.0.99.131) to 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "978d8b072e9d4bd49e227470e16ccfb1" committed_config { opid_index: 554 OBSOLETE_local: false peers { permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.194262 8021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aec442fecd64614858bce4a5bce5db4" candidate_uuid: "978d8b072e9d4bd49e227470e16ccfb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 554 } } ignore_live_leader: true dest_uuid: "40e8eacd02b44bb8bbec0a1dc205a671"
I20240503 13:59:59.194381 8021 raft_consensus.cc:3037] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 1 FOLLOWER]: Advancing to term 2
I20240503 13:59:59.195396 8021 raft_consensus.cc:2454] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 978d8b072e9d4bd49e227470e16ccfb1 in term 2.
I20240503 13:59:59.201705 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:59.201814 8268 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.205418 6889 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.210078 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.216734 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:59.217679 7020 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.227602 10703 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 13:59:59.244885 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.285367 10876 raft_consensus.cc:979] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: : Instructing follower 978d8b072e9d4bd49e227470e16ccfb1 to start an election
I20240503 13:59:59.285461 10876 raft_consensus.cc:1067] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 2 LEADER]: Signalling peer 978d8b072e9d4bd49e227470e16ccfb1 to start an election
I20240503 13:59:59.285938 10723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80d845691ee64bf3a5d6c2df59842a54"
dest_uuid: "978d8b072e9d4bd49e227470e16ccfb1"
from {username='slave'} at 127.0.99.134:37129
I20240503 13:59:59.286087 10723 raft_consensus.cc:482] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 13:59:59.286132 10723 raft_consensus.cc:3037] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Advancing to term 3
I20240503 13:59:59.286988 10723 raft_consensus.cc:504] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:59.287355 10723 leader_election.cc:288] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [CANDIDATE]: Term 3 election: Requested vote from peers 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579), 40e8eacd02b44bb8bbec0a1dc205a671 (127.0.99.132:41965)
I20240503 13:59:59.287652 8285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d845691ee64bf3a5d6c2df59842a54" candidate_uuid: "978d8b072e9d4bd49e227470e16ccfb1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "083b1c5c79a04af899eea33ba9f4502f"
I20240503 13:59:59.287755 8021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d845691ee64bf3a5d6c2df59842a54" candidate_uuid: "978d8b072e9d4bd49e227470e16ccfb1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "40e8eacd02b44bb8bbec0a1dc205a671"
I20240503 13:59:59.287806 8285 raft_consensus.cc:3032] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 13:59:59.287843 8285 raft_consensus.cc:729] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 083b1c5c79a04af899eea33ba9f4502f, State: Running, Role: LEADER
I20240503 13:59:59.287894 8021 raft_consensus.cc:3037] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 2 FOLLOWER]: Advancing to term 3
I20240503 13:59:59.288014 8285 consensus_queue.cc:260] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:59.288249 8285 raft_consensus.cc:3037] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 2 FOLLOWER]: Advancing to term 3
I20240503 13:59:59.288969 8021 raft_consensus.cc:2454] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 978d8b072e9d4bd49e227470e16ccfb1 in term 3.
I20240503 13:59:59.289301 10657 leader_election.cc:302] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40e8eacd02b44bb8bbec0a1dc205a671, 978d8b072e9d4bd49e227470e16ccfb1; no voters:
I20240503 13:59:59.289325 8285 raft_consensus.cc:2454] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 978d8b072e9d4bd49e227470e16ccfb1 in term 3.
I20240503 13:59:59.289480 11117 raft_consensus.cc:2781] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 13:59:59.289574 11117 raft_consensus.cc:686] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [term 3 LEADER]: Becoming Leader. State: Replica: 978d8b072e9d4bd49e227470e16ccfb1, State: Running, Role: LEADER
I20240503 13:59:59.289705 11117 consensus_queue.cc:237] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:59.291085 6633 catalog_manager.cc:5484] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 reported cstate change: term changed from 2 to 3, leader changed from 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) to 978d8b072e9d4bd49e227470e16ccfb1 (127.0.99.136). New cstate: current_term: 3 leader_uuid: "978d8b072e9d4bd49e227470e16ccfb1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.396445 8285 raft_consensus.cc:1261] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 2 FOLLOWER]: Refusing update from remote peer 978d8b072e9d4bd49e227470e16ccfb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 571. Preceding OpId from leader: term: 2 index: 572. (index mismatch)
I20240503 13:59:59.396956 11413 consensus_queue.cc:1035] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 572, Last known committed idx: 571, Time since last communication: 0.000s
I20240503 13:59:59.399737 6908 raft_consensus.cc:1261] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 2 FOLLOWER]: Refusing update from remote peer 978d8b072e9d4bd49e227470e16ccfb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 571. Preceding OpId from leader: term: 2 index: 572. (index mismatch)
I20240503 13:59:59.400166 11413 consensus_queue.cc:1035] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a58bb8fe574f5ab1485c59bf994570" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 572, Last known committed idx: 571, Time since last communication: 0.000s
I20240503 13:59:59.512894 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:59.513707 8268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:59.518097 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.518113 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.521731 10703 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:59.523303 7020 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.535979 6889 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.547011 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:59.577582 8154 consensus_queue.cc:237] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 621, Committed index: 621, Last appended: 2.621, Last appended by leader: 353, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 622 OBSOLETE_local: false peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:59.578667 10723 raft_consensus.cc:1261] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Refusing update from remote peer 4c5b7c28dd2a4920a2b4e97d1afe2384: Log matching property violated. Preceding OpId in replica: term: 2 index: 621. Preceding OpId from leader: term: 2 index: 622. (index mismatch)
I20240503 13:59:59.579108 10940 consensus_queue.cc:1035] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 622, Last known committed idx: 621, Time since last communication: 0.000s
I20240503 13:59:59.580917 11561 raft_consensus.cc:2932] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 [term 2 LEADER]: Committing config change with OpId 2.622: config changed from index 621 to 622, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 622 OBSOLETE_local: false peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.581357 10723 raft_consensus.cc:2932] T 57ea09e3f1f74d74839ddb0afd90acc2 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 FOLLOWER]: Committing config change with OpId 2.622: config changed from index 621 to 622, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New config: { opid_index: 622 OBSOLETE_local: false peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.583305 6632 catalog_manager.cc:5484] T 57ea09e3f1f74d74839ddb0afd90acc2 P 4c5b7c28dd2a4920a2b4e97d1afe2384 reported cstate change: config changed from index 621 to 622, VOTER 791cddb67b7e415aad7edb5151037418 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" committed_config { opid_index: 622 OBSOLETE_local: false peers { permanent_uuid: "4c5b7c28dd2a4920a2b4e97d1afe2384" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39967 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.585830 6758 tablet_service.cc:1514] Processing DeleteTablet for tablet 57ea09e3f1f74d74839ddb0afd90acc2 with delete_type TABLET_DATA_TOMBSTONED (TS 791cddb67b7e415aad7edb5151037418 not found in new config with opid_index 622) from {username='slave'} at 127.0.0.1:40392
I20240503 13:59:59.586210 11567 tablet_replica.cc:330] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418: stopping tablet replica
I20240503 13:59:59.586372 11567 raft_consensus.cc:2229] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.586575 11567 raft_consensus.cc:2258] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.588021 11567 ts_tablet_manager.cc:1875] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.590421 11567 ts_tablet_manager.cc:1888] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.621
I20240503 13:59:59.590512 11567 log.cc:1196] T 57ea09e3f1f74d74839ddb0afd90acc2 P 791cddb67b7e415aad7edb5151037418: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/57ea09e3f1f74d74839ddb0afd90acc2
I20240503 13:59:59.591084 6620 catalog_manager.cc:4830] TS 791cddb67b7e415aad7edb5151037418 (127.0.99.129:43893): tablet 57ea09e3f1f74d74839ddb0afd90acc2 (table rebalance_test_table_3 [id=bffd930039c544a68647ac9a405cf590]) successfully deleted
I20240503 13:59:59.602478 8021 raft_consensus.cc:1261] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 2 FOLLOWER]: Refusing update from remote peer 978d8b072e9d4bd49e227470e16ccfb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 554. Preceding OpId from leader: term: 2 index: 555. (index mismatch)
I20240503 13:59:59.602897 11413 consensus_queue.cc:1035] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 555, Last known committed idx: 554, Time since last communication: 0.000s
I20240503 13:59:59.605939 7038 raft_consensus.cc:1261] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Refusing update from remote peer 978d8b072e9d4bd49e227470e16ccfb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 554. Preceding OpId from leader: term: 2 index: 555. (index mismatch)
I20240503 13:59:59.606379 11117 consensus_queue.cc:1035] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c195eab0b34945afd10f3291ed080b" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 555, Last known committed idx: 554, Time since last communication: 0.000s
I20240503 13:59:59.641057 8268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:59.641703 7020 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.642499 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:59.650503 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.650678 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.651258 10703 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:59.652895 6889 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.671314 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:59.691780 8285 raft_consensus.cc:1261] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 3 FOLLOWER]: Refusing update from remote peer 978d8b072e9d4bd49e227470e16ccfb1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 13:59:59.692196 11117 consensus_queue.cc:1035] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.694759 8021 raft_consensus.cc:1261] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 3 FOLLOWER]: Refusing update from remote peer 978d8b072e9d4bd49e227470e16ccfb1: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 13:59:59.695397 11413 consensus_queue.cc:1035] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 13:59:59.705650 10723 consensus_queue.cc:237] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 572, Committed index: 572, Last appended: 2.572, Last appended by leader: 571, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 573 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:59.706545 8285 raft_consensus.cc:1261] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 2 FOLLOWER]: Refusing update from remote peer 978d8b072e9d4bd49e227470e16ccfb1: Log matching property violated. Preceding OpId in replica: term: 2 index: 572. Preceding OpId from leader: term: 2 index: 573. (index mismatch)
I20240503 13:59:59.706902 11413 consensus_queue.cc:1035] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 573, Last known committed idx: 572, Time since last communication: 0.000s
I20240503 13:59:59.708240 11117 raft_consensus.cc:2932] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 LEADER]: Committing config change with OpId 2.573: config changed from index 571 to 573, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 573 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.708657 8285 raft_consensus.cc:2932] T 4a9724b166dc4ca690fc912049779985 P 083b1c5c79a04af899eea33ba9f4502f [term 2 FOLLOWER]: Committing config change with OpId 2.573: config changed from index 571 to 573, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New config: { opid_index: 573 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.710568 6633 catalog_manager.cc:5484] T 4a9724b166dc4ca690fc912049779985 P 978d8b072e9d4bd49e227470e16ccfb1 reported cstate change: config changed from index 571 to 573, VOTER 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "978d8b072e9d4bd49e227470e16ccfb1" committed_config { opid_index: 573 OBSOLETE_local: false peers { permanent_uuid: "083b1c5c79a04af899eea33ba9f4502f" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 44579 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.713068 6889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a9724b166dc4ca690fc912049779985 with delete_type TABLET_DATA_TOMBSTONED (TS 52a58bb8fe574f5ab1485c59bf994570 not found in new config with opid_index 573) from {username='slave'} at 127.0.0.1:36310
I20240503 13:59:59.713411 11629 tablet_replica.cc:330] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: stopping tablet replica
I20240503 13:59:59.713544 11629 raft_consensus.cc:2229] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.713644 11629 raft_consensus.cc:2258] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.714927 11629 ts_tablet_manager.cc:1875] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.717060 11629 ts_tablet_manager.cc:1888] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.572
I20240503 13:59:59.717128 11629 log.cc:1196] T 4a9724b166dc4ca690fc912049779985 P 52a58bb8fe574f5ab1485c59bf994570: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4a9724b166dc4ca690fc912049779985
I20240503 13:59:59.717712 6620 catalog_manager.cc:4830] TS 52a58bb8fe574f5ab1485c59bf994570 (127.0.99.130:33439): tablet 4a9724b166dc4ca690fc912049779985 (table rebalance_test_table_1 [id=bb88829318084b1e8ec739465823de3c]) successfully deleted
I20240503 13:59:59.762936 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.764375 8268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:59.765395 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:59.776772 7020 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.778523 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.780871 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:59.787549 6889 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.798556 10703 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:59.833340 10723 consensus_queue.cc:237] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:59.834235 8021 raft_consensus.cc:1261] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 3 FOLLOWER]: Refusing update from remote peer 978d8b072e9d4bd49e227470e16ccfb1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 13:59:59.834561 11117 consensus_queue.cc:1035] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 13:59:59.835800 11413 raft_consensus.cc:2932] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.836159 8021 raft_consensus.cc:2932] T 80d845691ee64bf3a5d6c2df59842a54 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.838040 6632 catalog_manager.cc:5484] T 80d845691ee64bf3a5d6c2df59842a54 P 978d8b072e9d4bd49e227470e16ccfb1 reported cstate change: config changed from index 3 to 5, VOTER 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134) evicted. New cstate: current_term: 3 leader_uuid: "978d8b072e9d4bd49e227470e16ccfb1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.847407 8268 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d845691ee64bf3a5d6c2df59842a54 with delete_type TABLET_DATA_TOMBSTONED (TS 083b1c5c79a04af899eea33ba9f4502f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48796
I20240503 13:59:59.847853 11684 tablet_replica.cc:330] stopping tablet replica
I20240503 13:59:59.848171 11684 raft_consensus.cc:2229] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.848325 11684 raft_consensus.cc:2258] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.850167 11684 ts_tablet_manager.cc:1875] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 13:59:59.852439 11684 ts_tablet_manager.cc:1888] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240503 13:59:59.852526 11684 log.cc:1196] T 80d845691ee64bf3a5d6c2df59842a54 P 083b1c5c79a04af899eea33ba9f4502f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/80d845691ee64bf3a5d6c2df59842a54
I20240503 13:59:59.853044 6620 catalog_manager.cc:4830] TS 083b1c5c79a04af899eea33ba9f4502f (127.0.99.134:44579): tablet 80d845691ee64bf3a5d6c2df59842a54 (table TestTable [id=7284d6ec66314b6aa59c53dfab265168]) successfully deleted
I20240503 13:59:59.912285 8268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:59.915692 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 13:59:59.920116 7020 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.921892 6889 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 13:59:59.922410 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:59.925679 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.927829 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 13:59:59.956627 10703 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 13:59:59.992148 10723 consensus_queue.cc:237] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 555, Committed index: 555, Last appended: 2.555, Last appended by leader: 554, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 556 OBSOLETE_local: false peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } }
I20240503 13:59:59.993322 8021 raft_consensus.cc:1261] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 2 FOLLOWER]: Refusing update from remote peer 978d8b072e9d4bd49e227470e16ccfb1: Log matching property violated. Preceding OpId in replica: term: 2 index: 555. Preceding OpId from leader: term: 2 index: 556. (index mismatch)
I20240503 13:59:59.993743 11413 consensus_queue.cc:1035] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 556, Last known committed idx: 555, Time since last communication: 0.000s
I20240503 13:59:59.995325 11117 raft_consensus.cc:2932] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 [term 2 LEADER]: Committing config change with OpId 2.556: config changed from index 554 to 556, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 556 OBSOLETE_local: false peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.995775 8021 raft_consensus.cc:2932] T 4aec442fecd64614858bce4a5bce5db4 P 40e8eacd02b44bb8bbec0a1dc205a671 [term 2 FOLLOWER]: Committing config change with OpId 2.556: config changed from index 554 to 556, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New config: { opid_index: 556 OBSOLETE_local: false peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 13:59:59.997903 6635 catalog_manager.cc:5484] T 4aec442fecd64614858bce4a5bce5db4 P 978d8b072e9d4bd49e227470e16ccfb1 reported cstate change: config changed from index 554 to 556, VOTER 14c195eab0b34945afd10f3291ed080b (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "978d8b072e9d4bd49e227470e16ccfb1" committed_config { opid_index: 556 OBSOLETE_local: false peers { permanent_uuid: "40e8eacd02b44bb8bbec0a1dc205a671" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41965 } } peers { permanent_uuid: "978d8b072e9d4bd49e227470e16ccfb1" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 32963 } } }
I20240503 14:00:00.000411 7020 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aec442fecd64614858bce4a5bce5db4 with delete_type TABLET_DATA_TOMBSTONED (TS 14c195eab0b34945afd10f3291ed080b not found in new config with opid_index 556) from {username='slave'} at 127.0.0.1:42338
I20240503 14:00:00.000770 11735 tablet_replica.cc:330] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: stopping tablet replica
I20240503 14:00:00.000907 11735 raft_consensus.cc:2229] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:00.001013 11735 raft_consensus.cc:2258] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:00.002362 11735 ts_tablet_manager.cc:1875] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:00.004298 11735 ts_tablet_manager.cc:1888] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.555
I20240503 14:00:00.004372 11735 log.cc:1196] T 4aec442fecd64614858bce4a5bce5db4 P 14c195eab0b34945afd10f3291ed080b: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4aec442fecd64614858bce4a5bce5db4
I20240503 14:00:00.004947 6620 catalog_manager.cc:4830] TS 14c195eab0b34945afd10f3291ed080b (127.0.99.131:40699): tablet 4aec442fecd64614858bce4a5bce5db4 (table rebalance_test_table_4 [id=1b4f4315b2d34ad8a7a1bdd75ec294ef]) successfully deleted
I20240503 14:00:00.308522 8268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:00.310518 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:00.317389 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:00.318194 10703 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:00.324044 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:00.324586 6889 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:00.328024 7020 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:00.345664 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:00.463636 8268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:00.467368 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:00.478652 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:00.481329 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:00.481734 7020 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:00.487895 6889 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:00.490422 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:00.507674 10703 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:00.600332 7999 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:00.601972 8268 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:00.605069 7020 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:00.611160 8401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:00.613341 10703 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:00.614733 8136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:00.626511 6758 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:00.644742 6889 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
11a95c76ed294a80a8da622eab79df3a | 127.0.99.190:42439 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:45383 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
083b1c5c79a04af899eea33ba9f4502f | 127.0.99.134:44579 | HEALTHY | <none> | 0 | 0
14c195eab0b34945afd10f3291ed080b | 127.0.99.131:40699 | HEALTHY | <none> | 4 | 0
40e8eacd02b44bb8bbec0a1dc205a671 | 127.0.99.132:41965 | HEALTHY | <none> | 0 | 0
4c5b7c28dd2a4920a2b4e97d1afe2384 | 127.0.99.133:39967 | HEALTHY | <none> | 1 | 0
4f82135907754dd8b69d08d6b7eae847 | 127.0.99.135:34409 | HEALTHY | <none> | 1 | 0
52a58bb8fe574f5ab1485c59bf994570 | 127.0.99.130:33439 | HEALTHY | <none> | 4 | 0
791cddb67b7e415aad7edb5151037418 | 127.0.99.129:43893 | HEALTHY | <none> | 3 | 0
978d8b072e9d4bd49e227470e16ccfb1 | 127.0.99.136:32963 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:43893
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:33439
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:40699
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:41965
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:39967
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:44579
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:34409
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:32963
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:43893
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:33439
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:40699
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:41965
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:39967
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:44579
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:34409
server_dump_info_path | /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:32963
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:45383 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:00:00.665459 398 log_verifier.cc:126] Checking tablet 0a871746624e442597588274959d5e24
I20240503 14:00:00.753950 398 log_verifier.cc:177] Verified matching terms for 537 ops in tablet 0a871746624e442597588274959d5e24
I20240503 14:00:00.754274 398 log_verifier.cc:126] Checking tablet 4a9724b166dc4ca690fc912049779985
I20240503 14:00:00.846428 398 log_verifier.cc:177] Verified matching terms for 573 ops in tablet 4a9724b166dc4ca690fc912049779985
I20240503 14:00:00.846662 398 log_verifier.cc:126] Checking tablet 4aec442fecd64614858bce4a5bce5db4
I20240503 14:00:00.938572 398 log_verifier.cc:177] Verified matching terms for 556 ops in tablet 4aec442fecd64614858bce4a5bce5db4
I20240503 14:00:00.938746 398 log_verifier.cc:126] Checking tablet 4f882dbedd134f2998642ac747567db0
I20240503 14:00:01.032217 398 log_verifier.cc:177] Verified matching terms for 604 ops in tablet 4f882dbedd134f2998642ac747567db0
I20240503 14:00:01.032459 398 log_verifier.cc:126] Checking tablet 527c7e7cf4f24ce3bed276c1860aa041
I20240503 14:00:01.112136 398 log_verifier.cc:177] Verified matching terms for 514 ops in tablet 527c7e7cf4f24ce3bed276c1860aa041
I20240503 14:00:01.112315 398 log_verifier.cc:126] Checking tablet 57ea09e3f1f74d74839ddb0afd90acc2
I20240503 14:00:01.207204 398 log_verifier.cc:177] Verified matching terms for 622 ops in tablet 57ea09e3f1f74d74839ddb0afd90acc2
I20240503 14:00:01.207468 398 log_verifier.cc:126] Checking tablet 5fb5bbc56575455eb122d1e401fbdc64
I20240503 14:00:01.297281 398 log_verifier.cc:177] Verified matching terms for 575 ops in tablet 5fb5bbc56575455eb122d1e401fbdc64
I20240503 14:00:01.297540 398 log_verifier.cc:126] Checking tablet 6409f411c5d24e97b34aa9cce7c5a5ea
I20240503 14:00:01.377368 398 log_verifier.cc:177] Verified matching terms for 504 ops in tablet 6409f411c5d24e97b34aa9cce7c5a5ea
I20240503 14:00:01.377530 398 log_verifier.cc:126] Checking tablet 70583380c4e24b4694fc5477d880d944
I20240503 14:00:01.465837 398 log_verifier.cc:177] Verified matching terms for 572 ops in tablet 70583380c4e24b4694fc5477d880d944
I20240503 14:00:01.466032 398 log_verifier.cc:126] Checking tablet 80d845691ee64bf3a5d6c2df59842a54
I20240503 14:00:01.478401 398 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 80d845691ee64bf3a5d6c2df59842a54
I20240503 14:00:01.478480 398 log_verifier.cc:126] Checking tablet 9a52cb95109045d2bbfa3def26848224
I20240503 14:00:01.567431 398 log_verifier.cc:177] Verified matching terms for 560 ops in tablet 9a52cb95109045d2bbfa3def26848224
I20240503 14:00:01.567634 398 log_verifier.cc:126] Checking tablet a82b94a4baa84d778cc280d58c18af5b
I20240503 14:00:01.656698 398 log_verifier.cc:177] Verified matching terms for 578 ops in tablet a82b94a4baa84d778cc280d58c18af5b
I20240503 14:00:01.656872 398 log_verifier.cc:126] Checking tablet d118ea0cbd824762906095329740d0a3
I20240503 14:00:01.739957 398 log_verifier.cc:177] Verified matching terms for 535 ops in tablet d118ea0cbd824762906095329740d0a3
I20240503 14:00:01.740191 398 log_verifier.cc:126] Checking tablet e026fe8a768941ce84378f23b3dabff2
I20240503 14:00:01.838152 398 log_verifier.cc:177] Verified matching terms for 648 ops in tablet e026fe8a768941ce84378f23b3dabff2
I20240503 14:00:01.838366 398 log_verifier.cc:126] Checking tablet ef2d39830da24221a3dc44da57c0a6e8
I20240503 14:00:01.928792 398 log_verifier.cc:177] Verified matching terms for 578 ops in tablet ef2d39830da24221a3dc44da57c0a6e8
I20240503 14:00:01.928985 398 log_verifier.cc:126] Checking tablet f2b880960ed44b20b81ddf91c8534638
I20240503 14:00:02.022377 398 log_verifier.cc:177] Verified matching terms for 605 ops in tablet f2b880960ed44b20b81ddf91c8534638
I20240503 14:00:02.024003 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 6696
I20240503 14:00:02.033702 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 6827
I20240503 14:00:02.041430 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 6958
I20240503 14:00:02.048736 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 7940
I20240503 14:00:02.055117 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 8073
I20240503 14:00:02.061865 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 8205
I20240503 14:00:02.068167 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 8338
I20240503 14:00:02.074309 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 10640
I20240503 14:00:02.080847 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 6605
2024-05-03T14:00:02Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (33385 ms)
[----------] 1 test from RebalanceParamTest (33386 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240503 14:00:02.140424 398 test_util.cc:268] Using random seed: 584048896
I20240503 14:00:02.141233 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:00:02.141268 398 ts_itest-base.cc:113] --------------
I20240503 14:00:02.141287 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 14:00:02.141304 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:00:02.141320 398 ts_itest-base.cc:116] --------------
2024-05-03T14:00:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:00:02Z Disabled control of system clock
I20240503 14:00:02.150449 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44677
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:44677
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 14:00:02.259176 11864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:02.259438 11864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:02.259478 11864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:02.262423 11864 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:02.262481 11864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:00:02.262506 11864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:02.262522 11864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:00:02.262539 11864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:00:02.266234 11864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:44677
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.11864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:02.267118 11864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:02.268260 11864 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:02.276348 11869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:02.277545 11864 server_base.cc:1034] running on GCE node
W20240503 14:00:02.278306 11870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:02.279103 11872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:02.279675 11864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:02.280733 11864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:02.281922 11864 hybrid_clock.cc:648] HybridClock initialized: now 1714744802281899 us; error 40 us; skew 500 ppm
I20240503 14:00:02.284175 11864 webserver.cc:460] Webserver started at http://127.0.99.190:40631/ using document root <none> and password file <none>
I20240503 14:00:02.284911 11864 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:02.284991 11864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:02.285234 11864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:02.287137 11864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e301624eacf4431890d4c35d6eed9916"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.287626 11864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e301624eacf4431890d4c35d6eed9916"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.290838 11864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:00:02.293051 11878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.294065 11864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:00:02.294162 11864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e301624eacf4431890d4c35d6eed9916"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.294237 11864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:02.327533 11864 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:02.328166 11864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:02.328307 11864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:02.335050 11864 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:44677
I20240503 14:00:02.335062 11930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:44677 every 8 connection(s)
I20240503 14:00:02.336024 11864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:00:02.336206 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 11864
I20240503 14:00:02.336335 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:00:02.339121 11931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:02.344256 11931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916: Bootstrap starting.
I20240503 14:00:02.346484 11931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:02.347270 11931 log.cc:824] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:02.348838 11931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916: No bootstrap required, opened a new log
I20240503 14:00:02.351203 11931 raft_consensus.cc:348] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e301624eacf4431890d4c35d6eed9916" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44677 } }
I20240503 14:00:02.351421 11931 raft_consensus.cc:374] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:02.351491 11931 raft_consensus.cc:729] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e301624eacf4431890d4c35d6eed9916, State: Initialized, Role: FOLLOWER
I20240503 14:00:02.351902 11931 consensus_queue.cc:260] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e301624eacf4431890d4c35d6eed9916" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44677 } }
I20240503 14:00:02.352018 11931 raft_consensus.cc:388] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:00:02.352080 11931 raft_consensus.cc:482] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:00:02.352187 11931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:02.352937 11931 raft_consensus.cc:504] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e301624eacf4431890d4c35d6eed9916" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44677 } }
I20240503 14:00:02.353224 11931 leader_election.cc:302] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e301624eacf4431890d4c35d6eed9916; no voters:
I20240503 14:00:02.353475 11931 leader_election.cc:288] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:00:02.353564 11936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:02.354092 11936 raft_consensus.cc:686] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [term 1 LEADER]: Becoming Leader. State: Replica: e301624eacf4431890d4c35d6eed9916, State: Running, Role: LEADER
I20240503 14:00:02.354413 11936 consensus_queue.cc:237] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e301624eacf4431890d4c35d6eed9916" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44677 } }
I20240503 14:00:02.354559 11931 sys_catalog.cc:564] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:00:02.355952 11938 sys_catalog.cc:455] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e301624eacf4431890d4c35d6eed9916. Latest consensus state: current_term: 1 leader_uuid: "e301624eacf4431890d4c35d6eed9916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e301624eacf4431890d4c35d6eed9916" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44677 } } }
I20240503 14:00:02.355948 11937 sys_catalog.cc:455] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e301624eacf4431890d4c35d6eed9916" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e301624eacf4431890d4c35d6eed9916" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44677 } } }
I20240503 14:00:02.356096 11937 sys_catalog.cc:458] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [sys.catalog]: This master's current role is: LEADER
I20240503 14:00:02.356096 11938 sys_catalog.cc:458] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916 [sys.catalog]: This master's current role is: LEADER
I20240503 14:00:02.356448 11949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:00:02.359001 11949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:00:02.363760 11949 catalog_manager.cc:1347] Generated new cluster ID: cc0c95b23d964c4d9d5525a00aad802f
I20240503 14:00:02.363816 11949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:00:02.376405 11949 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:00:02.377256 11949 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:00:02.384253 11949 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e301624eacf4431890d4c35d6eed9916: Generated new TSK 0
I20240503 14:00:02.384905 11949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:00:02.394115 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:00:02.504778 11955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:02.505002 11955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:02.505048 11955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:02.508138 11955 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:02.508208 11955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:02.508296 11955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 14:00:02.511904 11955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.11955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:02.512780 11955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:02.513847 11955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:02.522850 11961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:02.523609 11963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:02.524394 11960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:02.525671 11955 server_base.cc:1034] running on GCE node
I20240503 14:00:02.526062 11955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:02.526616 11955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:02.527763 11955 hybrid_clock.cc:648] HybridClock initialized: now 1714744802527742 us; error 39 us; skew 500 ppm
I20240503 14:00:02.529737 11955 webserver.cc:460] Webserver started at http://127.0.99.129:36939/ using document root <none> and password file <none>
I20240503 14:00:02.530284 11955 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:02.530341 11955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:02.530527 11955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:02.532210 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "623e8dcb58544e558207d8e652dfed5f"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.532673 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "623e8dcb58544e558207d8e652dfed5f"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.535878 11955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:00:02.538013 11969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.538977 11955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:02.539086 11955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "623e8dcb58544e558207d8e652dfed5f"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.539167 11955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:02.553417 11955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:02.554036 11955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:02.554191 11955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:02.554675 11955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:02.555656 11955 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:02.555706 11955 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.555766 11955 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:02.555794 11955 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.563926 11955 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:36489
I20240503 14:00:02.563964 12082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:36489 every 8 connection(s)
I20240503 14:00:02.564822 11955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:00:02.570328 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 11955
I20240503 14:00:02.570462 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:00:02.572678 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:00:02.573541 12083 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:02.573772 12083 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:02.574321 12083 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:02.575783 11895 ts_manager.cc:194] Registered new tserver with Master: 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:02.577512 11895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:51829
W20240503 14:00:02.678602 12086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:02.678845 12086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:02.678889 12086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:02.681684 12086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:02.681741 12086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:02.681804 12086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 14:00:02.685186 12086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.12086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:02.685977 12086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:02.687064 12086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:02.695698 12091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:02.696739 12094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:02.697669 12092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:02.698499 12086 server_base.cc:1034] running on GCE node
I20240503 14:00:02.698891 12086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:02.699470 12086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:02.700616 12086 hybrid_clock.cc:648] HybridClock initialized: now 1714744802700594 us; error 45 us; skew 500 ppm
I20240503 14:00:02.702503 12086 webserver.cc:460] Webserver started at http://127.0.99.130:42145/ using document root <none> and password file <none>
I20240503 14:00:02.703076 12086 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:02.703137 12086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:02.703348 12086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:02.704984 12086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8b17d706d2344dbaae13cda9b0733143"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.705451 12086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8b17d706d2344dbaae13cda9b0733143"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.708808 12086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240503 14:00:02.710866 12100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.711836 12086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:00:02.711936 12086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8b17d706d2344dbaae13cda9b0733143"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.712023 12086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:02.737103 12086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:02.737862 12086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:02.738054 12086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:02.738700 12086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:02.739862 12086 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:02.739913 12086 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.739965 12086 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:02.740002 12086 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.750814 12086 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:36263
I20240503 14:00:02.750864 12213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:36263 every 8 connection(s)
I20240503 14:00:02.752004 12086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:00:02.760461 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 12086
I20240503 14:00:02.760597 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:00:02.761384 12214 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:02.761595 12214 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:02.762097 12214 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:02.762770 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:00:02.762949 11895 ts_manager.cc:194] Registered new tserver with Master: 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:02.763543 11895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:53881
W20240503 14:00:02.870021 12217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:02.870263 12217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:02.870332 12217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:02.873347 12217 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:02.873423 12217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:02.873512 12217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 14:00:02.876968 12217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.12217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:02.877863 12217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:02.879033 12217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:02.887794 12225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:02.888551 12222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:02.889317 12223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:02.890563 12217 server_base.cc:1034] running on GCE node
I20240503 14:00:02.890978 12217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:02.891572 12217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:02.892732 12217 hybrid_clock.cc:648] HybridClock initialized: now 1714744802892725 us; error 27 us; skew 500 ppm
I20240503 14:00:02.894591 12217 webserver.cc:460] Webserver started at http://127.0.99.131:44479/ using document root <none> and password file <none>
I20240503 14:00:02.895138 12217 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:02.895198 12217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:02.895401 12217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:02.897126 12217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ca5386f84994496fa5376624391728ee"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.897624 12217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ca5386f84994496fa5376624391728ee"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.900862 12217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:00:02.902859 12231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.903832 12217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:02.903939 12217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ca5386f84994496fa5376624391728ee"
format_stamp: "Formatted at 2024-05-03 14:00:02 on dist-test-slave-snql"
I20240503 14:00:02.904032 12217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:02.930255 12217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:02.930862 12217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:02.931026 12217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:02.931587 12217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:02.932442 12217 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:02.932498 12217 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.932557 12217 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:02.932585 12217 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:02.940557 12217 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:36399
I20240503 14:00:02.940579 12344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:36399 every 8 connection(s)
I20240503 14:00:02.941475 12217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:00:02.949119 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 12217
I20240503 14:00:02.949265 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:00:02.949793 12345 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:02.950021 12345 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:02.950455 12345 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:02.951354 11895 ts_manager.cc:194] Registered new tserver with Master: ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:02.951426 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:00:02.951920 11895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:52587
W20240503 14:00:03.057713 12348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:03.057914 12348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:03.057955 12348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:03.060964 12348 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:03.061030 12348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:03.061093 12348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 14:00:03.064628 12348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.12348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:03.065464 12348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:03.066671 12348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:03.075510 12353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.076671 12356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.077520 12354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:03.078418 12348 server_base.cc:1034] running on GCE node
I20240503 14:00:03.078776 12348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:03.079349 12348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:03.080505 12348 hybrid_clock.cc:648] HybridClock initialized: now 1714744803080492 us; error 39 us; skew 500 ppm
I20240503 14:00:03.082392 12348 webserver.cc:460] Webserver started at http://127.0.99.132:38007/ using document root <none> and password file <none>
I20240503 14:00:03.082937 12348 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:03.083006 12348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:03.083174 12348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:03.084820 12348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "84c63c6f551949f6a1d7c07760824361"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.085294 12348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "84c63c6f551949f6a1d7c07760824361"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.088528 12348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 14:00:03.090787 12362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.091784 12348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:00:03.091894 12348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "84c63c6f551949f6a1d7c07760824361"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.091980 12348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:03.108189 12348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:03.108775 12348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:03.108940 12348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:03.109457 12348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:03.110312 12348 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:03.110361 12348 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.110420 12348 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:03.110450 12348 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.118661 12348 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:33927
I20240503 14:00:03.118711 12475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:33927 every 8 connection(s)
I20240503 14:00:03.119627 12348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:00:03.127547 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 12348
I20240503 14:00:03.127697 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:00:03.127840 12476 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:03.128043 12476 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:03.128464 12476 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:03.129374 11895 ts_manager.cc:194] Registered new tserver with Master: 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:03.129899 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:00:03.130091 11895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:50327
W20240503 14:00:03.241202 12479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:03.241430 12479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:03.241499 12479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:03.244520 12479 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:03.244592 12479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:03.244685 12479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 14:00:03.248185 12479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.12479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:03.249061 12479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:03.250155 12479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:03.259496 12485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.260255 12484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.261384 12487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:03.262286 12479 server_base.cc:1034] running on GCE node
I20240503 14:00:03.262634 12479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:03.263170 12479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:03.264328 12479 hybrid_clock.cc:648] HybridClock initialized: now 1714744803264314 us; error 29 us; skew 500 ppm
I20240503 14:00:03.266201 12479 webserver.cc:460] Webserver started at http://127.0.99.133:46713/ using document root <none> and password file <none>
I20240503 14:00:03.266785 12479 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:03.266840 12479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:03.267026 12479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:03.268658 12479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "66190085d1c04e068d5c0dec9c4bf07d"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.269147 12479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "66190085d1c04e068d5c0dec9c4bf07d"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.272408 12479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 14:00:03.274533 12493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.275555 12479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:00:03.275647 12479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "66190085d1c04e068d5c0dec9c4bf07d"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.275740 12479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:03.292758 12479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:03.293318 12479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:03.293462 12479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:03.293954 12479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:03.294792 12479 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:03.294842 12479 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.294915 12479 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:03.294945 12479 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.303120 12479 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:37005
I20240503 14:00:03.303140 12606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:37005 every 8 connection(s)
I20240503 14:00:03.304018 12479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:00:03.305900 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 12479
I20240503 14:00:03.306006 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:00:03.308267 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:00:03.313410 12607 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:03.313622 12607 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:03.314062 12607 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:03.314958 11895 ts_manager.cc:194] Registered new tserver with Master: 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005)
I20240503 14:00:03.315809 11895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:55931
W20240503 14:00:03.415098 12610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:03.415402 12610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:03.415503 12610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:03.418474 12610 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:03.418535 12610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:03.418598 12610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 14:00:03.422106 12610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.12610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:03.422928 12610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:03.424055 12610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:03.432770 12615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.433934 12616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.434839 12618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:03.435778 12610 server_base.cc:1034] running on GCE node
I20240503 14:00:03.436141 12610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:03.436668 12610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:03.437824 12610 hybrid_clock.cc:648] HybridClock initialized: now 1714744803437805 us; error 38 us; skew 500 ppm
I20240503 14:00:03.440017 12610 webserver.cc:460] Webserver started at http://127.0.99.134:34509/ using document root <none> and password file <none>
I20240503 14:00:03.440608 12610 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:03.440671 12610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:03.440878 12610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:03.442654 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "b67f337e10994234baedbf0b19c13513"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.443151 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "b67f337e10994234baedbf0b19c13513"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.446408 12610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:03.448565 12624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.449581 12610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:03.449685 12610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "b67f337e10994234baedbf0b19c13513"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.449772 12610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:03.460740 12610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:03.461263 12610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:03.461413 12610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:03.461898 12610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:03.462724 12610 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:03.462771 12610 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.462829 12610 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:03.462857 12610 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.470937 12610 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:37075
I20240503 14:00:03.470961 12737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:37075 every 8 connection(s)
I20240503 14:00:03.471892 12610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:00:03.474831 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 12610
I20240503 14:00:03.474956 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:00:03.477268 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:00:03.481375 12738 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:03.481637 12738 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:03.482225 12738 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:03.483269 11895 ts_manager.cc:194] Registered new tserver with Master: b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075)
I20240503 14:00:03.483791 11895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:48831
I20240503 14:00:03.580361 12083 heartbeater.cc:503] Master 127.0.99.190:44677 was elected leader, sending a full tablet report...
W20240503 14:00:03.588148 12741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:03.588375 12741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:03.588445 12741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:03.591409 12741 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:03.591475 12741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:03.591562 12741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 14:00:03.595225 12741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.12741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:03.596207 12741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:03.597360 12741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:03.606287 12749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.607174 12746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.608218 12747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:03.609237 12741 server_base.cc:1034] running on GCE node
I20240503 14:00:03.609618 12741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:03.610168 12741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:03.611295 12741 hybrid_clock.cc:648] HybridClock initialized: now 1714744803611286 us; error 30 us; skew 500 ppm
I20240503 14:00:03.613226 12741 webserver.cc:460] Webserver started at http://127.0.99.135:39327/ using document root <none> and password file <none>
I20240503 14:00:03.613744 12741 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:03.613796 12741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:03.613965 12741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:03.615770 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.616789 12741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.620014 12741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:03.622138 12755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.623106 12741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:03.623210 12741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.623313 12741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:03.653180 12741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:03.653836 12741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:03.654012 12741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:03.654601 12741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:03.655534 12741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:03.655583 12741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.655624 12741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:03.655647 12741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.663678 12741 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:46845
I20240503 14:00:03.663728 12868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:46845 every 8 connection(s)
I20240503 14:00:03.664600 12741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:00:03.672921 12869 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:03.673135 12869 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:03.673543 12869 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:03.674280 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 12741
I20240503 14:00:03.674400 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:00:03.674454 11895 ts_manager.cc:194] Registered new tserver with Master: e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
I20240503 14:00:03.675024 11895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:43515
I20240503 14:00:03.676473 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:00:03.765360 12214 heartbeater.cc:503] Master 127.0.99.190:44677 was elected leader, sending a full tablet report...
W20240503 14:00:03.793187 12872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:03.793380 12872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:03.793438 12872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:03.796377 12872 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:03.796437 12872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:03.796527 12872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 14:00:03.800069 12872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.12872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:03.800901 12872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:03.801947 12872 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:03.810793 12877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.811472 12878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:03.812294 12880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:03.813521 12872 server_base.cc:1034] running on GCE node
I20240503 14:00:03.813870 12872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:03.814394 12872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:03.815543 12872 hybrid_clock.cc:648] HybridClock initialized: now 1714744803815532 us; error 31 us; skew 500 ppm
I20240503 14:00:03.817590 12872 webserver.cc:460] Webserver started at http://127.0.99.136:40097/ using document root <none> and password file <none>
I20240503 14:00:03.818143 12872 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:03.818203 12872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:03.818373 12872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:03.820120 12872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "1b4a61e8a11440e7a9f4b300d2442f75"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.820571 12872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "1b4a61e8a11440e7a9f4b300d2442f75"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.823858 12872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:00:03.825860 12886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.826875 12872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:03.826972 12872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1b4a61e8a11440e7a9f4b300d2442f75"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:03.827049 12872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:03.843936 12872 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:03.844503 12872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:03.844653 12872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:03.845146 12872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:03.845992 12872 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:03.846043 12872 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.846104 12872 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:03.846130 12872 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:03.854269 12872 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:34763
I20240503 14:00:03.854298 12999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:34763 every 8 connection(s)
I20240503 14:00:03.855193 12872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:00:03.862519 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 12872
I20240503 14:00:03.862661 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:00:03.863399 13000 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:03.863617 13000 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:03.864046 13000 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:03.864886 11894 ts_manager.cc:194] Registered new tserver with Master: 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763)
I20240503 14:00:03.865381 11894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:37783
I20240503 14:00:03.876579 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:00:03.885650 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:00:03.885735 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 12479
I20240503 14:00:03.891633 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 12610
I20240503 14:00:03.897015 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 12741
I20240503 14:00:03.902871 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu with pid 12872
I20240503 14:00:03.953981 12345 heartbeater.cc:503] Master 127.0.99.190:44677 was elected leader, sending a full tablet report...
I20240503 14:00:04.132223 12476 heartbeater.cc:503] Master 127.0.99.190:44677 was elected leader, sending a full tablet report...
I20240503 14:00:07.659895 11894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51640:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:00:07.679739 12147 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6ca408c4254678b67a74f49a34881b (DEFAULT_TABLE table=rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:07.679739 12278 tablet_service.cc:1467] Processing CreateTablet for tablet f5c06806a19e4f4592d26ed4acc651cd (DEFAULT_TABLE table=rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:07.679739 12409 tablet_service.cc:1467] Processing CreateTablet for tablet f5c06806a19e4f4592d26ed4acc651cd (DEFAULT_TABLE table=rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:07.679739 12410 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4f3bc64e7145e2b692beab730d1496 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:07.680724 12147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6ca408c4254678b67a74f49a34881b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.680724 12278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c06806a19e4f4592d26ed4acc651cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.680727 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4f3bc64e7145e2b692beab730d1496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.681310 12279 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6ca408c4254678b67a74f49a34881b (DEFAULT_TABLE table=rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:07.681545 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6ca408c4254678b67a74f49a34881b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.682282 12148 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4f3bc64e7145e2b692beab730d1496 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:07.682519 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4f3bc64e7145e2b692beab730d1496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.682909 12409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c06806a19e4f4592d26ed4acc651cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.685340 12017 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4f3bc64e7145e2b692beab730d1496 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:07.686175 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4f3bc64e7145e2b692beab730d1496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.688319 13020 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:07.690194 13020 tablet_bootstrap.cc:654] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:07.690454 13022 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:07.691002 12015 tablet_service.cc:1467] Processing CreateTablet for tablet f5c06806a19e4f4592d26ed4acc651cd (DEFAULT_TABLE table=rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:07.691144 13021 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:07.691278 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c06806a19e4f4592d26ed4acc651cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.691731 12016 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6ca408c4254678b67a74f49a34881b (DEFAULT_TABLE table=rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:07.691990 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6ca408c4254678b67a74f49a34881b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:07.692377 13022 tablet_bootstrap.cc:654] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:07.693140 13022 log.cc:824] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:07.693346 13021 tablet_bootstrap.cc:654] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:07.693468 13023 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:07.694067 13021 log.cc:824] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:07.694505 13020 log.cc:824] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:07.694901 13022 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:07.695066 13022 ts_tablet_manager.cc:1367] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:00:07.695693 13023 tablet_bootstrap.cc:654] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:07.696581 13023 log.cc:824] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:07.696687 13020 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:07.696854 13020 ts_tablet_manager.cc:1367] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:00:07.698956 13021 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:07.699108 13021 ts_tablet_manager.cc:1367] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:00:07.699182 13023 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:07.699362 13023 ts_tablet_manager.cc:1367] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 14:00:07.699654 13022 raft_consensus.cc:348] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.699859 13022 raft_consensus.cc:374] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:07.699946 13022 raft_consensus.cc:729] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:07.700444 13022 consensus_queue.cc:260] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.701366 13022 ts_tablet_manager.cc:1398] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:00:07.701690 13022 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:07.702222 13021 raft_consensus.cc:348] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.702761 13022 tablet_bootstrap.cc:654] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:07.702855 13023 raft_consensus.cc:348] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.703100 13023 raft_consensus.cc:374] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:07.703190 13023 raft_consensus.cc:729] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:07.703708 13023 consensus_queue.cc:260] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.704075 13022 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:07.704146 13022 ts_tablet_manager.cc:1367] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:07.704564 13023 ts_tablet_manager.cc:1398] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240503 14:00:07.704672 13022 raft_consensus.cc:348] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:07.704751 13022 raft_consensus.cc:374] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:07.704777 13022 raft_consensus.cc:729] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:07.704877 13023 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:07.704881 13022 consensus_queue.cc:260] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:07.704926 13020 raft_consensus.cc:348] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.705098 13020 raft_consensus.cc:374] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:07.705117 13022 ts_tablet_manager.cc:1398] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:07.705144 13020 raft_consensus.cc:729] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:07.705364 13021 raft_consensus.cc:374] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:07.705447 13021 raft_consensus.cc:729] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:07.705765 13023 tablet_bootstrap.cc:654] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:07.705823 13021 consensus_queue.cc:260] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.706110 13020 consensus_queue.cc:260] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.706753 13021 ts_tablet_manager.cc:1398] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20240503 14:00:07.706957 13020 ts_tablet_manager.cc:1398] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.010s user 0.000s sys 0.004s
I20240503 14:00:07.707345 13020 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:07.707399 13023 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:07.707475 13023 ts_tablet_manager.cc:1367] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:00:07.707981 13023 raft_consensus.cc:348] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.708115 13023 raft_consensus.cc:374] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:07.708159 13023 raft_consensus.cc:729] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:07.708276 13023 consensus_queue.cc:260] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.708479 13020 tablet_bootstrap.cc:654] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:07.708590 13023 ts_tablet_manager.cc:1398] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:07.708750 13023 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:07.709590 13023 tablet_bootstrap.cc:654] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:07.709934 13021 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:07.710884 13021 tablet_bootstrap.cc:654] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:07.711520 13020 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:07.711941 13023 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:07.712020 13023 ts_tablet_manager.cc:1367] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:00:07.712306 13021 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:07.712375 13021 ts_tablet_manager.cc:1367] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:00:07.712409 13023 raft_consensus.cc:348] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:07.712496 13023 raft_consensus.cc:374] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:07.712535 13023 raft_consensus.cc:729] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:07.712653 13023 consensus_queue.cc:260] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:07.712846 13020 ts_tablet_manager.cc:1367] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:00:07.712883 13023 ts_tablet_manager.cc:1398] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:07.712904 13021 raft_consensus.cc:348] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:07.713027 13021 raft_consensus.cc:374] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:07.713059 13021 raft_consensus.cc:729] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:07.713179 13021 consensus_queue.cc:260] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:07.713384 13020 raft_consensus.cc:348] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.713482 13020 raft_consensus.cc:374] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:07.713491 13021 ts_tablet_manager.cc:1398] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:07.713521 13020 raft_consensus.cc:729] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:07.713646 13020 consensus_queue.cc:260] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.714314 13020 ts_tablet_manager.cc:1398] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 14:00:07.756772 12215 tablet.cc:2369] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:00:07.814699 13030 raft_consensus.cc:482] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:07.814944 13030 raft_consensus.cc:504] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.816257 13030 leader_election.cc:288] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
W20240503 14:00:07.819569 12084 tablet.cc:2369] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:00:07.820127 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:07.820441 12168 raft_consensus.cc:2454] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:07.820442 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:07.820693 12430 raft_consensus.cc:2454] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:07.820895 11973 leader_election.cc:302] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:07.821172 13030 raft_consensus.cc:2781] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:07.821240 13030 raft_consensus.cc:482] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:07.821269 13030 raft_consensus.cc:3037] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:07.822160 13030 raft_consensus.cc:504] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.822512 13030 leader_election.cc:288] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:07.822806 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:07.822798 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:07.822916 12168 raft_consensus.cc:3037] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:07.822916 12430 raft_consensus.cc:3037] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:07.823969 12430 raft_consensus.cc:2454] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:07.823969 12168 raft_consensus.cc:2454] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:07.824244 11970 leader_election.cc:302] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:07.824400 13030 raft_consensus.cc:2781] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:07.824594 13030 raft_consensus.cc:686] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:07.824882 13030 consensus_queue.cc:237] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.827636 11894 catalog_manager.cc:5484] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:07.848963 13030 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:07.849102 13030 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:07.849601 13030 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:07.850109 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:07.850271 12430 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:07.850572 11970 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:07.850752 13030 raft_consensus.cc:2781] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:07.850831 13030 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:07.850876 13030 raft_consensus.cc:3037] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:07.851080 13039 raft_consensus.cc:482] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:07.851182 13039 raft_consensus.cc:504] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.851568 13039 leader_election.cc:288] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:07.851944 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6ca408c4254678b67a74f49a34881b" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:07.852011 13030 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:07.852074 12168 raft_consensus.cc:2454] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:07.852315 11973 leader_election.cc:302] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:07.852322 13030 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:07.852474 13030 raft_consensus.cc:2781] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:07.852509 13030 raft_consensus.cc:482] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:07.852548 13030 raft_consensus.cc:3037] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:07.852675 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:07.852795 12430 raft_consensus.cc:3037] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:07.853410 13030 raft_consensus.cc:504] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.853596 12430 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:07.853745 13030 leader_election.cc:288] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:07.854161 11970 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:07.854136 12299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:07.854113 12297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:07.854362 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6ca408c4254678b67a74f49a34881b" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:07.854455 12296 raft_consensus.cc:3037] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:07.854490 12299 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:07.854116 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6ca408c4254678b67a74f49a34881b" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:07.854915 13030 raft_consensus.cc:2781] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:07.854909 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6ca408c4254678b67a74f49a34881b" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:07.854998 13030 raft_consensus.cc:686] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:07.855024 12168 raft_consensus.cc:3037] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:07.855130 13030 consensus_queue.cc:237] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:07.855652 12296 raft_consensus.cc:2454] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:07.855921 12168 raft_consensus.cc:2454] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:07.856405 11973 leader_election.cc:302] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:07.856581 13039 raft_consensus.cc:2781] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:07.856659 13039 raft_consensus.cc:686] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:07.856767 13039 consensus_queue.cc:237] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:07.857017 11894 catalog_manager.cc:5484] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:07.859989 11894 catalog_manager.cc:5484] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
W20240503 14:00:07.875056 12477 tablet.cc:2369] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:00:07.946364 12346 tablet.cc:2369] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:00:07.971434 13043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:07.971685 13043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:07.974613 13043 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 14:00:08.056049 12430 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:08.056049 12168 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:08.056954 13039 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:08.057368 13039 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:08.057715 12430 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:08.058460 13030 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:08.058573 12299 raft_consensus.cc:3037] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.059818 12299 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:08.060784 13030 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:08.062450 12167 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:08.062606 12296 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:08.063390 13030 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:08.063684 13030 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:08.073493 13070 mvcc.cc:205] Tried to move back new op lower bound from 7023594733793058816 to 7023594732982579200. Current Snapshot: MvccSnapshot[applied={T|T < 7023594733793058816}]
I20240503 14:00:08.075402 13076 mvcc.cc:205] Tried to move back new op lower bound from 7023594733792165888 to 7023594732976058368. Current Snapshot: MvccSnapshot[applied={T|T < 7023594733792165888}]
I20240503 14:00:08.076445 13083 mvcc.cc:205] Tried to move back new op lower bound from 7023594733792165888 to 7023594732976058368. Current Snapshot: MvccSnapshot[applied={T|T < 7023594733792165888}]
I20240503 14:00:08.087097 13073 mvcc.cc:205] Tried to move back new op lower bound from 7023594733789339648 to 7023594732853043200. Current Snapshot: MvccSnapshot[applied={T|T < 7023594733789339648}]
INSERT report
rows total: 2000
time total: 93.6295 ms
time per row: 0.0468148 ms
I20240503 14:00:08.115226 11887 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51640:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:00:08.127869 12278 tablet_service.cc:1467] Processing CreateTablet for tablet a0106f7a40cb467eb4738890ac4e22d4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.128021 12410 tablet_service.cc:1467] Processing CreateTablet for tablet c2e9f6eecbec45988b290775277dbddc (DEFAULT_TABLE table=rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.128244 12278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0106f7a40cb467eb4738890ac4e22d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.128324 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e9f6eecbec45988b290775277dbddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.128448 12148 tablet_service.cc:1467] Processing CreateTablet for tablet a0106f7a40cb467eb4738890ac4e22d4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.128690 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0106f7a40cb467eb4738890ac4e22d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.130175 13022 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:08.130625 12279 tablet_service.cc:1467] Processing CreateTablet for tablet f4f9cb87c70e43dbb0d850c275ebb8a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.130736 13021 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:08.130829 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f9cb87c70e43dbb0d850c275ebb8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.130892 12409 tablet_service.cc:1467] Processing CreateTablet for tablet a0106f7a40cb467eb4738890ac4e22d4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.131067 13022 tablet_bootstrap.cc:654] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.131106 12409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0106f7a40cb467eb4738890ac4e22d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.131367 13020 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:08.131543 13021 tablet_bootstrap.cc:654] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.132146 13020 tablet_bootstrap.cc:654] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.132236 12015 tablet_service.cc:1467] Processing CreateTablet for tablet f4f9cb87c70e43dbb0d850c275ebb8a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.132508 13021 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:08.132501 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f9cb87c70e43dbb0d850c275ebb8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.132581 13021 ts_tablet_manager.cc:1367] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:08.132989 13021 raft_consensus.cc:348] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:08.133092 13021 raft_consensus.cc:374] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.133129 13021 raft_consensus.cc:729] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.133250 13021 consensus_queue.cc:260] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:08.133529 13021 ts_tablet_manager.cc:1398] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:08.133723 12278 tablet_service.cc:1467] Processing CreateTablet for tablet c2e9f6eecbec45988b290775277dbddc (DEFAULT_TABLE table=rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.133929 12278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e9f6eecbec45988b290775277dbddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.133910 12147 tablet_service.cc:1467] Processing CreateTablet for tablet f4f9cb87c70e43dbb0d850c275ebb8a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.134114 12147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f9cb87c70e43dbb0d850c275ebb8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.134879 12148 tablet_service.cc:1467] Processing CreateTablet for tablet c2e9f6eecbec45988b290775277dbddc (DEFAULT_TABLE table=rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.135087 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e9f6eecbec45988b290775277dbddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.135772 13023 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:08.136602 13021 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:08.136638 13023 tablet_bootstrap.cc:654] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.137499 13021 tablet_bootstrap.cc:654] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.137786 13023 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:08.137868 13023 ts_tablet_manager.cc:1367] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:00:08.138316 13023 raft_consensus.cc:348] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.138425 13023 raft_consensus.cc:374] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.138463 13023 raft_consensus.cc:729] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.138613 13023 consensus_queue.cc:260] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.138882 13023 ts_tablet_manager.cc:1398] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 14:00:08.139050 13020 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:08.139123 13020 ts_tablet_manager.cc:1367] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 14:00:08.139632 13020 raft_consensus.cc:348] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.139734 13020 raft_consensus.cc:374] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.139767 13020 raft_consensus.cc:729] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.139906 13020 consensus_queue.cc:260] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.140059 13021 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:08.140123 13021 ts_tablet_manager.cc:1367] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:00:08.140177 13020 ts_tablet_manager.cc:1398] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:08.140357 13020 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:08.140530 13021 raft_consensus.cc:348] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.140635 13021 raft_consensus.cc:374] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.140672 13021 raft_consensus.cc:729] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.140779 13021 consensus_queue.cc:260] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.141016 13021 ts_tablet_manager.cc:1398] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:08.141242 13020 tablet_bootstrap.cc:654] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.142832 13022 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:08.142927 13022 ts_tablet_manager.cc:1367] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 14:00:08.143414 13022 raft_consensus.cc:348] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.143504 13022 raft_consensus.cc:374] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.143544 13022 raft_consensus.cc:729] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.143671 13022 consensus_queue.cc:260] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.143880 13022 ts_tablet_manager.cc:1398] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:08.144047 13022 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:08.144714 13022 tablet_bootstrap.cc:654] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.145820 13031 raft_consensus.cc:482] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:08.145916 13031 raft_consensus.cc:504] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:08.147001 13031 leader_election.cc:288] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:08.148216 13022 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:08.148305 13022 ts_tablet_manager.cc:1367] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:00:08.148715 13022 raft_consensus.cc:348] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.148828 13022 raft_consensus.cc:374] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.148872 13022 raft_consensus.cc:729] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.149333 13020 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:08.149401 13020 ts_tablet_manager.cc:1367] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240503 14:00:08.149713 13020 raft_consensus.cc:348] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.149782 13020 raft_consensus.cc:374] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.149821 13020 raft_consensus.cc:729] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.149960 13022 consensus_queue.cc:260] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.150137 13020 consensus_queue.cc:260] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.150502 13022 ts_tablet_manager.cc:1398] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:08.151751 13020 ts_tablet_manager.cc:1398] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:00:08.151914 13020 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:08.152159 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e9f6eecbec45988b290775277dbddc" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:08.152206 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e9f6eecbec45988b290775277dbddc" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:08.152843 13020 tablet_bootstrap.cc:654] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
W20240503 14:00:08.153014 12366 leader_election.cc:341] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:00:08.153211 12366 leader_election.cc:341] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:00:08.153276 12366 leader_election.cc:302] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 8b17d706d2344dbaae13cda9b0733143, ca5386f84994496fa5376624391728ee
I20240503 14:00:08.153496 13031 raft_consensus.cc:2726] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:00:08.153645 13022 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:08.154208 13020 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:08.154294 13020 ts_tablet_manager.cc:1367] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:00:08.154645 13020 raft_consensus.cc:348] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:08.154739 13020 raft_consensus.cc:374] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.154778 13020 raft_consensus.cc:729] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.154876 13022 tablet_bootstrap.cc:654] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.154901 13020 consensus_queue.cc:260] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:08.155122 13020 ts_tablet_manager.cc:1398] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:08.156088 13022 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:08.156173 13022 ts_tablet_manager.cc:1367] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:00:08.156575 13022 raft_consensus.cc:348] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:08.156657 13022 raft_consensus.cc:374] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.156688 13022 raft_consensus.cc:729] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.156806 13022 consensus_queue.cc:260] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:08.157052 13022 ts_tablet_manager.cc:1398] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:08.195307 13028 raft_consensus.cc:482] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:08.195447 13028 raft_consensus.cc:504] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:08.196527 13028 leader_election.cc:288] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:08.200451 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e9f6eecbec45988b290775277dbddc" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:08.200474 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e9f6eecbec45988b290775277dbddc" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:08.200613 12168 raft_consensus.cc:2454] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:08.200626 12430 raft_consensus.cc:2454] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:08.200942 12235 leader_election.cc:302] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:08.201162 13028 raft_consensus.cc:2781] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:08.201233 13028 raft_consensus.cc:482] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:08.201287 13028 raft_consensus.cc:3037] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.202159 13028 raft_consensus.cc:504] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:08.202488 13028 leader_election.cc:288] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Requested vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:08.202765 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e9f6eecbec45988b290775277dbddc" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:08.202801 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2e9f6eecbec45988b290775277dbddc" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:08.202867 12430 raft_consensus.cc:3037] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.202898 12168 raft_consensus.cc:3037] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.203750 12430 raft_consensus.cc:2454] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:08.203754 12168 raft_consensus.cc:2454] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:08.204005 12232 leader_election.cc:302] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:08.204170 13028 raft_consensus.cc:2781] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:08.204346 13028 raft_consensus.cc:686] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:08.204634 13028 consensus_queue.cc:237] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:08.206445 11891 catalog_manager.cc:5484] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:08.219414 13033 raft_consensus.cc:482] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:08.219532 13033 raft_consensus.cc:504] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.220504 13033 leader_election.cc:288] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:08.225342 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f9cb87c70e43dbb0d850c275ebb8a5" candidate_uuid: "8b17d706d2344dbaae13cda9b0733143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:08.225395 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f9cb87c70e43dbb0d850c275ebb8a5" candidate_uuid: "8b17d706d2344dbaae13cda9b0733143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:08.225503 12296 raft_consensus.cc:2454] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b17d706d2344dbaae13cda9b0733143 in term 0.
I20240503 14:00:08.225585 12037 raft_consensus.cc:2454] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b17d706d2344dbaae13cda9b0733143 in term 0.
I20240503 14:00:08.225971 12104 leader_election.cc:302] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:08.226233 13033 raft_consensus.cc:2781] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:08.226301 13033 raft_consensus.cc:482] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:08.226349 13033 raft_consensus.cc:3037] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.227483 13033 raft_consensus.cc:504] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.227876 13033 leader_election.cc:288] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 1 election: Requested vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:08.228250 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f9cb87c70e43dbb0d850c275ebb8a5" candidate_uuid: "8b17d706d2344dbaae13cda9b0733143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:08.228385 12296 raft_consensus.cc:3037] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.229302 12296 raft_consensus.cc:2454] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b17d706d2344dbaae13cda9b0733143 in term 1.
I20240503 14:00:08.229545 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f9cb87c70e43dbb0d850c275ebb8a5" candidate_uuid: "8b17d706d2344dbaae13cda9b0733143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:08.229590 12104 leader_election.cc:302] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:08.229692 12037 raft_consensus.cc:3037] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.229838 13033 raft_consensus.cc:2781] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:08.230037 13033 raft_consensus.cc:686] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Becoming Leader. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Running, Role: LEADER
I20240503 14:00:08.230331 13033 consensus_queue.cc:237] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.230593 12037 raft_consensus.cc:2454] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b17d706d2344dbaae13cda9b0733143 in term 1.
I20240503 14:00:08.232267 11891 catalog_manager.cc:5484] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:00:08.433118 13028 raft_consensus.cc:482] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:08.433280 13028 raft_consensus.cc:504] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.433746 13028 leader_election.cc:288] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:08.434173 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0106f7a40cb467eb4738890ac4e22d4" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:08.434181 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0106f7a40cb467eb4738890ac4e22d4" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:08.434304 12168 raft_consensus.cc:2454] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:08.434331 12430 raft_consensus.cc:2454] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:08.434600 12235 leader_election.cc:302] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:08.434772 13028 raft_consensus.cc:2781] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:08.434827 13028 raft_consensus.cc:482] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:08.434890 13028 raft_consensus.cc:3037] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.435812 13028 raft_consensus.cc:504] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.436117 13028 leader_election.cc:288] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:08.436422 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0106f7a40cb467eb4738890ac4e22d4" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:08.436475 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0106f7a40cb467eb4738890ac4e22d4" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:08.436542 12168 raft_consensus.cc:3037] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.436594 12430 raft_consensus.cc:3037] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.437460 12168 raft_consensus.cc:2454] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:08.437460 12430 raft_consensus.cc:2454] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:08.437747 12232 leader_election.cc:302] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:08.437922 13028 raft_consensus.cc:2781] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:08.438014 13028 raft_consensus.cc:686] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:08.438176 13028 consensus_queue.cc:237] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.439723 11891 catalog_manager.cc:5484] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
W20240503 14:00:08.605937 13122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:08.606173 13122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:08.609306 13122 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 14:00:08.656246 13028 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:08.664039 13115 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:08.672235 13119 consensus_queue.cc:1035] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:08.676760 13033 consensus_queue.cc:1035] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:08.696600 12168 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:08.697358 12429 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:08.698118 13028 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:08.698813 13115 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.848 ms
time per row: 0.0604239 ms
I20240503 14:00:08.770793 11892 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51640:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:00:08.776669 12409 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2e1517a8ac4041bfb4b90634cb0fa6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.777022 12409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2e1517a8ac4041bfb4b90634cb0fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.777034 12015 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2e1517a8ac4041bfb4b90634cb0fa6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.777112 12017 tablet_service.cc:1467] Processing CreateTablet for tablet 11a8c270532947ba9b505aace44b56f6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.777307 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2e1517a8ac4041bfb4b90634cb0fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.777570 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11a8c270532947ba9b505aace44b56f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.777766 12408 tablet_service.cc:1467] Processing CreateTablet for tablet 11a8c270532947ba9b505aace44b56f6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.778012 12408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11a8c270532947ba9b505aace44b56f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.779507 12410 tablet_service.cc:1467] Processing CreateTablet for tablet d1e9155230c74e92b6ee441de713a419 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.779735 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e9155230c74e92b6ee441de713a419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.779902 12016 tablet_service.cc:1467] Processing CreateTablet for tablet d1e9155230c74e92b6ee441de713a419 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.780134 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e9155230c74e92b6ee441de713a419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.780269 13179 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:08.781250 13179 tablet_bootstrap.cc:654] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.781473 13180 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:08.782317 13180 tablet_bootstrap.cc:654] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.782280 12278 tablet_service.cc:1467] Processing CreateTablet for tablet d1e9155230c74e92b6ee441de713a419 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.782562 12278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e9155230c74e92b6ee441de713a419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.783293 13180 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:08.783442 13180 ts_tablet_manager.cc:1367] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:08.783913 13179 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:08.783881 13180 raft_consensus.cc:348] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.784019 13180 raft_consensus.cc:374] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.784096 13180 raft_consensus.cc:729] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.784116 13179 ts_tablet_manager.cc:1367] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:00:08.784283 13180 consensus_queue.cc:260] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.784641 13180 ts_tablet_manager.cc:1398] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:08.784655 13179 raft_consensus.cc:348] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.784845 13179 raft_consensus.cc:374] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.784880 13180 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:08.784929 13179 raft_consensus.cc:729] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.785123 13179 consensus_queue.cc:260] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.785301 12148 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2e1517a8ac4041bfb4b90634cb0fa6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.785575 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2e1517a8ac4041bfb4b90634cb0fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.785732 13180 tablet_bootstrap.cc:654] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.786195 13179 ts_tablet_manager.cc:1398] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:00:08.786401 13179 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:08.787652 12147 tablet_service.cc:1467] Processing CreateTablet for tablet 11a8c270532947ba9b505aace44b56f6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:08.787889 12147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11a8c270532947ba9b505aace44b56f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:08.787950 13180 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:08.788029 13185 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:08.788060 13180 ts_tablet_manager.cc:1367] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:08.788467 13179 tablet_bootstrap.cc:654] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.788455 13180 raft_consensus.cc:348] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.788590 13180 raft_consensus.cc:374] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.788676 13180 raft_consensus.cc:729] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.788822 13180 consensus_queue.cc:260] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.789013 13185 tablet_bootstrap.cc:654] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.789168 13180 ts_tablet_manager.cc:1398] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:08.789361 13180 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:08.789953 13179 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:08.790040 13179 ts_tablet_manager.cc:1367] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:00:08.790158 13180 tablet_bootstrap.cc:654] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.790525 13179 raft_consensus.cc:348] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.790621 13179 raft_consensus.cc:374] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.790653 13179 raft_consensus.cc:729] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.790771 13179 consensus_queue.cc:260] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.791043 13179 ts_tablet_manager.cc:1398] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:08.791210 13179 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:08.791355 13183 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:08.792095 13179 tablet_bootstrap.cc:654] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.792181 13180 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:08.792259 13180 ts_tablet_manager.cc:1367] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:00:08.792697 13180 raft_consensus.cc:348] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.792754 13183 tablet_bootstrap.cc:654] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.792795 13180 raft_consensus.cc:374] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.792836 13180 raft_consensus.cc:729] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.792966 13180 consensus_queue.cc:260] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.793234 13180 ts_tablet_manager.cc:1398] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:08.793741 13185 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:08.793836 13185 ts_tablet_manager.cc:1367] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:00:08.793689 13179 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:08.794026 13179 ts_tablet_manager.cc:1367] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:08.794196 13185 raft_consensus.cc:348] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.794271 13185 raft_consensus.cc:374] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.794291 13185 raft_consensus.cc:729] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.794399 13185 consensus_queue.cc:260] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:08.794523 13179 raft_consensus.cc:348] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.794636 13179 raft_consensus.cc:374] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.794659 13185 ts_tablet_manager.cc:1398] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:08.794674 13179 raft_consensus.cc:729] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.794812 13185 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:08.794811 13179 consensus_queue.cc:260] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.795375 13179 ts_tablet_manager.cc:1398] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:08.795658 13185 tablet_bootstrap.cc:654] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:08.796756 13185 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:08.796834 13185 ts_tablet_manager.cc:1367] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:08.797230 13185 raft_consensus.cc:348] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.797299 13185 raft_consensus.cc:374] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.797282 13183 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:08.797320 13185 raft_consensus.cc:729] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.797379 13183 ts_tablet_manager.cc:1367] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:00:08.797500 13185 consensus_queue.cc:260] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.797782 13185 ts_tablet_manager.cc:1398] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:08.797871 13183 raft_consensus.cc:348] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.797991 13183 raft_consensus.cc:374] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:08.798066 13183 raft_consensus.cc:729] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:08.798224 13183 consensus_queue.cc:260] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.798542 13183 ts_tablet_manager.cc:1398] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:08.898929 13177 raft_consensus.cc:482] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:08.899065 13177 raft_consensus.cc:504] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.899602 13177 leader_election.cc:288] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:08.900091 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e9155230c74e92b6ee441de713a419" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:08.900255 12430 raft_consensus.cc:2454] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:08.900559 12232 leader_election.cc:302] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:08.900743 13177 raft_consensus.cc:2781] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:08.900810 13177 raft_consensus.cc:482] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:08.900844 13177 raft_consensus.cc:3037] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.901768 13177 raft_consensus.cc:504] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.902117 13177 leader_election.cc:288] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Requested vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:08.902474 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e9155230c74e92b6ee441de713a419" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:08.902590 12430 raft_consensus.cc:3037] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.903497 12430 raft_consensus.cc:2454] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:08.903633 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e9155230c74e92b6ee441de713a419" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:08.903673 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e9155230c74e92b6ee441de713a419" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:08.903769 12037 raft_consensus.cc:2454] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:08.903944 12232 leader_election.cc:302] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:08.904088 13177 raft_consensus.cc:2781] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:08.904162 13177 raft_consensus.cc:686] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:08.904295 13177 consensus_queue.cc:237] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.905994 11893 catalog_manager.cc:5484] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:08.913964 13145 raft_consensus.cc:482] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:08.914079 13145 raft_consensus.cc:504] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.914445 13145 leader_election.cc:288] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:08.914888 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a8c270532947ba9b505aace44b56f6" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:08.915028 12168 raft_consensus.cc:2454] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:08.915391 12366 leader_election.cc:302] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:08.915575 13145 raft_consensus.cc:2781] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:08.915649 13145 raft_consensus.cc:482] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:08.915702 13145 raft_consensus.cc:3037] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.916604 13145 raft_consensus.cc:504] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.916944 13145 leader_election.cc:288] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:08.917261 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a8c270532947ba9b505aace44b56f6" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:08.917363 12168 raft_consensus.cc:3037] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:08.918237 12168 raft_consensus.cc:2454] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 1.
I20240503 14:00:08.918507 12366 leader_election.cc:302] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:08.919194 13145 raft_consensus.cc:2781] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:08.919437 13145 raft_consensus.cc:686] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Becoming Leader. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Running, Role: LEADER
I20240503 14:00:08.919463 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a8c270532947ba9b505aace44b56f6" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:08.919507 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11a8c270532947ba9b505aace44b56f6" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:08.919579 12036 raft_consensus.cc:2454] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:08.919783 13145 consensus_queue.cc:237] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:08.921658 11893 catalog_manager.cc:5484] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84c63c6f551949f6a1d7c07760824361 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:09.078464 13089 raft_consensus.cc:482] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:09.078603 13089 raft_consensus.cc:504] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:09.078976 13089 leader_election.cc:288] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:09.079479 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2e1517a8ac4041bfb4b90634cb0fa6" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:09.079479 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2e1517a8ac4041bfb4b90634cb0fa6" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:09.079648 12430 raft_consensus.cc:2454] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:09.079638 12168 raft_consensus.cc:2454] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:09.079967 11970 leader_election.cc:302] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:09.080138 13089 raft_consensus.cc:2781] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:09.080197 13089 raft_consensus.cc:482] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:09.080226 13089 raft_consensus.cc:3037] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.081064 13089 raft_consensus.cc:504] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:09.081370 13089 leader_election.cc:288] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:09.081662 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2e1517a8ac4041bfb4b90634cb0fa6" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:09.081673 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2e1517a8ac4041bfb4b90634cb0fa6" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:09.081759 12168 raft_consensus.cc:3037] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.081771 12430 raft_consensus.cc:3037] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.082511 12430 raft_consensus.cc:2454] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:09.082512 12168 raft_consensus.cc:2454] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:09.082852 11970 leader_election.cc:302] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:09.083072 13089 raft_consensus.cc:2781] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:09.083293 13089 raft_consensus.cc:686] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:09.083444 13089 consensus_queue.cc:237] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:09.085069 11893 catalog_manager.cc:5484] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
W20240503 14:00:09.252080 13201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:09.252306 13201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:09.255430 13201 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:00:09.298465 13145 consensus_queue.cc:1035] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:09.301690 12037 raft_consensus.cc:3037] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.303181 13196 consensus_queue.cc:1035] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:09.311122 12037 raft_consensus.cc:3037] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.312533 13177 consensus_queue.cc:1035] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:09.315873 13028 consensus_queue.cc:1035] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:09.342355 12168 raft_consensus.cc:1261] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:09.343369 13199 consensus_queue.cc:1035] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:09.344058 12429 raft_consensus.cc:1261] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:09.346208 13199 consensus_queue.cc:1035] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.038 ms
time per row: 0.0520189 ms
I20240503 14:00:09.400125 11893 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51640:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:00:09.405555 12278 tablet_service.cc:1467] Processing CreateTablet for tablet adf2c44328e244779955e712b99f8331 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.405555 12147 tablet_service.cc:1467] Processing CreateTablet for tablet adf2c44328e244779955e712b99f8331 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.405869 12147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf2c44328e244779955e712b99f8331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.405844 12279 tablet_service.cc:1467] Processing CreateTablet for tablet 211de6f35a844e278e8c1eb1621e7c5f (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.405869 12278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf2c44328e244779955e712b99f8331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.406314 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211de6f35a844e278e8c1eb1621e7c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.406448 12277 tablet_service.cc:1467] Processing CreateTablet for tablet 4a608f683c4f41f3bbdfc9b4c219ce49 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.406675 12277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a608f683c4f41f3bbdfc9b4c219ce49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.406845 12408 tablet_service.cc:1467] Processing CreateTablet for tablet 4a608f683c4f41f3bbdfc9b4c219ce49 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.407137 12408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a608f683c4f41f3bbdfc9b4c219ce49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.408344 12410 tablet_service.cc:1467] Processing CreateTablet for tablet 211de6f35a844e278e8c1eb1621e7c5f (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.408561 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211de6f35a844e278e8c1eb1621e7c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.408906 13264 tablet_bootstrap.cc:492] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:09.409446 12016 tablet_service.cc:1467] Processing CreateTablet for tablet adf2c44328e244779955e712b99f8331 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.409703 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf2c44328e244779955e712b99f8331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.409901 13264 tablet_bootstrap.cc:654] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.410594 12015 tablet_service.cc:1467] Processing CreateTablet for tablet 4a608f683c4f41f3bbdfc9b4c219ce49 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.410822 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a608f683c4f41f3bbdfc9b4c219ce49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.411183 12147 tablet_service.cc:1467] Processing CreateTablet for tablet 211de6f35a844e278e8c1eb1621e7c5f (DEFAULT_TABLE table=rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.411473 12147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211de6f35a844e278e8c1eb1621e7c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.412304 13263 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:09.412458 13265 tablet_bootstrap.cc:492] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:09.413198 13263 tablet_bootstrap.cc:654] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.413415 13265 tablet_bootstrap.cc:654] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.413848 13264 tablet_bootstrap.cc:492] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:09.413954 13267 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:09.413985 13264 ts_tablet_manager.cc:1367] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:00:09.414379 13263 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:09.414443 13265 tablet_bootstrap.cc:492] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:09.414490 13263 ts_tablet_manager.cc:1367] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:09.414508 13265 ts_tablet_manager.cc:1367] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:09.414659 13264 raft_consensus.cc:348] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.414861 13264 raft_consensus.cc:374] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.414884 13267 tablet_bootstrap.cc:654] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.414961 13264 raft_consensus.cc:729] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.414949 13265 raft_consensus.cc:348] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.414953 13263 raft_consensus.cc:348] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:09.415053 13265 raft_consensus.cc:374] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.415089 13265 raft_consensus.cc:729] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.415089 13263 raft_consensus.cc:374] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.415124 13263 raft_consensus.cc:729] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.415197 13264 consensus_queue.cc:260] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.415270 13263 consensus_queue.cc:260] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:09.415573 13264 ts_tablet_manager.cc:1398] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:09.415648 13263 ts_tablet_manager.cc:1398] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:09.415748 13264 tablet_bootstrap.cc:492] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:09.415791 13263 tablet_bootstrap.cc:492] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:09.416749 13264 tablet_bootstrap.cc:654] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.416931 13267 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:09.417014 13267 ts_tablet_manager.cc:1367] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:09.415215 13265 consensus_queue.cc:260] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.417411 13263 tablet_bootstrap.cc:654] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.417536 13265 ts_tablet_manager.cc:1398] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:00:09.417498 13267 raft_consensus.cc:348] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:09.417610 13267 raft_consensus.cc:374] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.417658 13267 raft_consensus.cc:729] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.417718 13265 tablet_bootstrap.cc:492] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:09.417783 13267 consensus_queue.cc:260] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:09.418069 13267 ts_tablet_manager.cc:1398] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:09.418195 13267 tablet_bootstrap.cc:492] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:09.418581 13265 tablet_bootstrap.cc:654] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.419045 13267 tablet_bootstrap.cc:654] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.420871 13263 tablet_bootstrap.cc:492] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:09.420967 13263 ts_tablet_manager.cc:1367] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:00:09.421218 13267 tablet_bootstrap.cc:492] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:09.421326 13267 ts_tablet_manager.cc:1367] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:09.421466 13263 raft_consensus.cc:348] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.421572 13263 raft_consensus.cc:374] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.421623 13263 raft_consensus.cc:729] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.421690 13267 raft_consensus.cc:348] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.421780 13267 raft_consensus.cc:374] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.421752 13263 consensus_queue.cc:260] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.421808 13267 raft_consensus.cc:729] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.421916 13264 tablet_bootstrap.cc:492] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:09.421903 13267 consensus_queue.cc:260] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.421988 13264 ts_tablet_manager.cc:1367] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:00:09.422011 13263 ts_tablet_manager.cc:1398] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:09.422163 13267 ts_tablet_manager.cc:1398] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:09.422354 13264 raft_consensus.cc:348] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.422469 13264 raft_consensus.cc:374] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.422502 13264 raft_consensus.cc:729] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.422616 13264 consensus_queue.cc:260] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.422883 13264 ts_tablet_manager.cc:1398] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:09.423143 13264 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:09.423152 13265 tablet_bootstrap.cc:492] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:09.423269 13265 ts_tablet_manager.cc:1367] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:00:09.423954 13264 tablet_bootstrap.cc:654] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.423959 13265 raft_consensus.cc:348] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.424593 13265 raft_consensus.cc:374] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.424685 13265 raft_consensus.cc:729] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.424865 13265 consensus_queue.cc:260] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.425117 13265 ts_tablet_manager.cc:1398] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:09.425806 13264 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:09.425909 13264 ts_tablet_manager.cc:1367] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:09.426337 13264 raft_consensus.cc:348] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:09.426442 13264 raft_consensus.cc:374] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.426474 13264 raft_consensus.cc:729] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.426573 13264 consensus_queue.cc:260] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:09.426800 13264 ts_tablet_manager.cc:1398] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:09.438336 13199 raft_consensus.cc:482] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:09.438462 13199 raft_consensus.cc:504] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.438843 13199 leader_election.cc:288] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:09.439220 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a608f683c4f41f3bbdfc9b4c219ce49" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:09.439220 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a608f683c4f41f3bbdfc9b4c219ce49" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:09.439373 12296 raft_consensus.cc:2454] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:09.439373 12429 raft_consensus.cc:2454] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:09.439646 11973 leader_election.cc:302] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:09.439834 13199 raft_consensus.cc:2781] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:09.439904 13199 raft_consensus.cc:482] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:09.439949 13199 raft_consensus.cc:3037] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.440829 13199 raft_consensus.cc:504] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.441164 13199 leader_election.cc:288] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:09.441429 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a608f683c4f41f3bbdfc9b4c219ce49" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:09.441555 12429 raft_consensus.cc:3037] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.441586 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a608f683c4f41f3bbdfc9b4c219ce49" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:09.441704 12296 raft_consensus.cc:3037] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.442617 12429 raft_consensus.cc:2454] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:09.442617 12296 raft_consensus.cc:2454] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:09.442871 11970 leader_election.cc:302] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:09.443038 13199 raft_consensus.cc:2781] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:09.443120 13199 raft_consensus.cc:686] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:09.443298 13199 consensus_queue.cc:237] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.444844 11894 catalog_manager.cc:5484] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:09.534056 13245 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:09.534191 13245 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.534583 13245 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:09.535018 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211de6f35a844e278e8c1eb1621e7c5f" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:09.535092 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211de6f35a844e278e8c1eb1621e7c5f" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:09.535154 12296 raft_consensus.cc:2454] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:09.535279 12168 raft_consensus.cc:2454] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:09.535513 12366 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:09.535679 13245 raft_consensus.cc:2781] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:09.535734 13245 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:09.535759 13245 raft_consensus.cc:3037] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.536579 13245 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.536863 13245 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Requested vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:09.537150 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211de6f35a844e278e8c1eb1621e7c5f" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:09.537227 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211de6f35a844e278e8c1eb1621e7c5f" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:09.537266 12296 raft_consensus.cc:3037] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.537318 12168 raft_consensus.cc:3037] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.537999 12296 raft_consensus.cc:2454] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 1.
I20240503 14:00:09.538005 12168 raft_consensus.cc:2454] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 1.
I20240503 14:00:09.538247 12366 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:09.538380 13245 raft_consensus.cc:2781] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:09.538466 13245 raft_consensus.cc:686] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Becoming Leader. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Running, Role: LEADER
I20240503 14:00:09.538633 13245 consensus_queue.cc:237] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.540316 11894 catalog_manager.cc:5484] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84c63c6f551949f6a1d7c07760824361 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:09.548652 13262 raft_consensus.cc:482] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:09.548768 13262 raft_consensus.cc:504] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:09.549110 13262 leader_election.cc:288] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:09.549504 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf2c44328e244779955e712b99f8331" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:09.549573 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf2c44328e244779955e712b99f8331" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:09.549625 12168 raft_consensus.cc:2454] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:09.549718 12037 raft_consensus.cc:2454] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:09.549911 12235 leader_election.cc:302] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:09.550089 13262 raft_consensus.cc:2781] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:09.550145 13262 raft_consensus.cc:482] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:09.550196 13262 raft_consensus.cc:3037] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.550992 13262 raft_consensus.cc:504] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:09.551321 13262 leader_election.cc:288] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:09.551642 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf2c44328e244779955e712b99f8331" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:09.551672 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf2c44328e244779955e712b99f8331" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:09.551752 12168 raft_consensus.cc:3037] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.551764 12037 raft_consensus.cc:3037] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.552441 12168 raft_consensus.cc:2454] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:09.552498 12037 raft_consensus.cc:2454] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:09.552748 12235 leader_election.cc:302] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:09.552902 13262 raft_consensus.cc:2781] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:09.552979 13262 raft_consensus.cc:686] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:09.553112 13262 consensus_queue.cc:237] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:09.554615 11894 catalog_manager.cc:5484] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
W20240503 14:00:09.667044 13279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:09.667341 13279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:09.670264 13279 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 14:00:09.741521 12168 raft_consensus.cc:1261] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:09.745195 12037 raft_consensus.cc:1261] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:09.745291 13145 consensus_queue.cc:1035] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:09.746076 13262 consensus_queue.cc:1035] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:09.746186 12296 raft_consensus.cc:1261] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:09.746919 13245 consensus_queue.cc:1035] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:09.747045 12168 raft_consensus.cc:1261] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:09.750483 13028 consensus_queue.cc:1035] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:09.751227 12429 raft_consensus.cc:1261] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:09.751684 13199 consensus_queue.cc:1035] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:09.751941 12299 raft_consensus.cc:1261] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:09.754060 13089 consensus_queue.cc:1035] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.003 ms
time per row: 0.0515016 ms
I20240503 14:00:09.814316 11887 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51640:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:00:09.819605 12147 tablet_service.cc:1467] Processing CreateTablet for tablet c42fee9fadbb4601a7fe32eb6a04e767 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.819950 12147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42fee9fadbb4601a7fe32eb6a04e767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.820495 12278 tablet_service.cc:1467] Processing CreateTablet for tablet c42fee9fadbb4601a7fe32eb6a04e767 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.820817 12278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42fee9fadbb4601a7fe32eb6a04e767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.821049 12410 tablet_service.cc:1467] Processing CreateTablet for tablet faa0a6ca00d442eaaf9fca7fc1259ec7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.821120 12408 tablet_service.cc:1467] Processing CreateTablet for tablet a6a89230883347d4a64d96d321247012 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.821301 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa0a6ca00d442eaaf9fca7fc1259ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.821595 12408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a89230883347d4a64d96d321247012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.822444 12015 tablet_service.cc:1467] Processing CreateTablet for tablet c42fee9fadbb4601a7fe32eb6a04e767 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.822712 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42fee9fadbb4601a7fe32eb6a04e767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.823071 12016 tablet_service.cc:1467] Processing CreateTablet for tablet faa0a6ca00d442eaaf9fca7fc1259ec7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.823297 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa0a6ca00d442eaaf9fca7fc1259ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.823472 13265 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:09.823577 12146 tablet_service.cc:1467] Processing CreateTablet for tablet a6a89230883347d4a64d96d321247012 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.823794 12146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a89230883347d4a64d96d321247012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.824374 13265 tablet_bootstrap.cc:654] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.824821 13263 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:09.825378 12148 tablet_service.cc:1467] Processing CreateTablet for tablet faa0a6ca00d442eaaf9fca7fc1259ec7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.825610 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa0a6ca00d442eaaf9fca7fc1259ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.825659 13263 tablet_bootstrap.cc:654] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.825994 13265 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:09.826063 13265 ts_tablet_manager.cc:1367] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:00:09.826282 13267 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:09.826504 13265 raft_consensus.cc:348] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.826607 13265 raft_consensus.cc:374] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.826651 13265 raft_consensus.cc:729] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.826784 13265 consensus_queue.cc:260] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.827086 13267 tablet_bootstrap.cc:654] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.827086 13265 ts_tablet_manager.cc:1398] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:09.827304 13265 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:09.827381 13264 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:09.828205 13264 tablet_bootstrap.cc:654] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.828563 13265 tablet_bootstrap.cc:654] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.820503 12279 tablet_service.cc:1467] Processing CreateTablet for tablet a6a89230883347d4a64d96d321247012 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:09.829262 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a89230883347d4a64d96d321247012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:09.829291 13263 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:09.829375 13263 ts_tablet_manager.cc:1367] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:00:09.829614 13267 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:09.829691 13267 ts_tablet_manager.cc:1367] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:00:09.829818 13263 raft_consensus.cc:348] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.829906 13263 raft_consensus.cc:374] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.829936 13263 raft_consensus.cc:729] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.830051 13263 consensus_queue.cc:260] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.830127 13267 raft_consensus.cc:348] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.830211 13267 raft_consensus.cc:374] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.830241 13267 raft_consensus.cc:729] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.830327 13263 ts_tablet_manager.cc:1398] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:09.830354 13267 consensus_queue.cc:260] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.830494 13263 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:09.831358 13263 tablet_bootstrap.cc:654] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.832113 13267 ts_tablet_manager.cc:1398] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:00:09.832298 13267 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:09.832568 13263 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:09.832654 13263 ts_tablet_manager.cc:1367] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:00:09.832851 13265 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:09.832993 13265 ts_tablet_manager.cc:1367] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:00:09.833077 13263 raft_consensus.cc:348] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.833178 13263 raft_consensus.cc:374] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.833214 13263 raft_consensus.cc:729] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.833323 13263 consensus_queue.cc:260] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.833503 13265 raft_consensus.cc:348] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.833607 13265 raft_consensus.cc:374] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.833611 13263 ts_tablet_manager.cc:1398] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:09.833649 13265 raft_consensus.cc:729] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.833748 13263 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:09.833781 13265 consensus_queue.cc:260] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.834046 13265 ts_tablet_manager.cc:1398] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:09.834583 13263 tablet_bootstrap.cc:654] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.835666 13267 tablet_bootstrap.cc:654] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.836864 13264 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:09.836864 13267 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:09.836941 13267 ts_tablet_manager.cc:1367] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:00:09.836941 13264 ts_tablet_manager.cc:1367] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 14:00:09.837319 13267 raft_consensus.cc:348] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.837319 13264 raft_consensus.cc:348] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.837417 13264 raft_consensus.cc:374] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.837417 13267 raft_consensus.cc:374] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.837451 13264 raft_consensus.cc:729] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.837455 13267 raft_consensus.cc:729] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.837560 13267 consensus_queue.cc:260] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.837560 13264 consensus_queue.cc:260] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.837813 13264 ts_tablet_manager.cc:1398] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:09.837949 13264 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:09.838245 13267 ts_tablet_manager.cc:1398] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:09.838709 13264 tablet_bootstrap.cc:654] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:09.839942 13263 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:09.840015 13263 ts_tablet_manager.cc:1367] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 14:00:09.840442 13263 raft_consensus.cc:348] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.840559 13263 raft_consensus.cc:374] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.840590 13263 raft_consensus.cc:729] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.840657 13264 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:09.840715 13263 consensus_queue.cc:260] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.840766 13264 ts_tablet_manager.cc:1367] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:09.840991 13263 ts_tablet_manager.cc:1398] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:09.841208 13264 raft_consensus.cc:348] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.841315 13264 raft_consensus.cc:374] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:09.841365 13264 raft_consensus.cc:729] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:09.841475 13264 consensus_queue.cc:260] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.841749 13264 ts_tablet_manager.cc:1398] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:09.881798 13163 raft_consensus.cc:482] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:09.881919 13163 raft_consensus.cc:504] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.882262 13163 leader_election.cc:288] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:09.882577 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42fee9fadbb4601a7fe32eb6a04e767" candidate_uuid: "8b17d706d2344dbaae13cda9b0733143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:09.882673 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42fee9fadbb4601a7fe32eb6a04e767" candidate_uuid: "8b17d706d2344dbaae13cda9b0733143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:09.882730 12037 raft_consensus.cc:2454] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b17d706d2344dbaae13cda9b0733143 in term 0.
I20240503 14:00:09.882818 12298 raft_consensus.cc:2454] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b17d706d2344dbaae13cda9b0733143 in term 0.
I20240503 14:00:09.883013 12104 leader_election.cc:302] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:09.883194 13163 raft_consensus.cc:2781] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:09.883291 13163 raft_consensus.cc:482] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:09.883340 13163 raft_consensus.cc:3037] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.884271 13163 raft_consensus.cc:504] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.884552 13163 leader_election.cc:288] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 1 election: Requested vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:09.884846 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42fee9fadbb4601a7fe32eb6a04e767" candidate_uuid: "8b17d706d2344dbaae13cda9b0733143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:09.884909 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42fee9fadbb4601a7fe32eb6a04e767" candidate_uuid: "8b17d706d2344dbaae13cda9b0733143" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:09.884960 12037 raft_consensus.cc:3037] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.885010 12298 raft_consensus.cc:3037] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.885949 12298 raft_consensus.cc:2454] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b17d706d2344dbaae13cda9b0733143 in term 1.
I20240503 14:00:09.885949 12037 raft_consensus.cc:2454] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b17d706d2344dbaae13cda9b0733143 in term 1.
I20240503 14:00:09.886201 12104 leader_election.cc:302] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:09.886361 13163 raft_consensus.cc:2781] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:09.886541 13163 raft_consensus.cc:686] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Becoming Leader. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Running, Role: LEADER
I20240503 14:00:09.886669 13163 consensus_queue.cc:237] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:09.888211 11887 catalog_manager.cc:5484] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:09.950748 13089 raft_consensus.cc:482] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:09.950899 13089 raft_consensus.cc:504] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.951335 13089 leader_election.cc:288] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:09.951711 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa0a6ca00d442eaaf9fca7fc1259ec7" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:09.951711 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa0a6ca00d442eaaf9fca7fc1259ec7" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:09.951865 12429 raft_consensus.cc:2454] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:09.951865 12167 raft_consensus.cc:2454] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:09.952188 11973 leader_election.cc:302] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:09.952370 13089 raft_consensus.cc:2781] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:09.952435 13089 raft_consensus.cc:482] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:09.952466 13089 raft_consensus.cc:3037] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.953464 13089 raft_consensus.cc:504] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.953789 13089 leader_election.cc:288] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:09.954085 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa0a6ca00d442eaaf9fca7fc1259ec7" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:09.954120 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa0a6ca00d442eaaf9fca7fc1259ec7" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:09.954188 12429 raft_consensus.cc:3037] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.954228 12167 raft_consensus.cc:3037] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:09.955006 12429 raft_consensus.cc:2454] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:09.955173 12167 raft_consensus.cc:2454] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:09.955322 11970 leader_election.cc:302] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:09.955511 13089 raft_consensus.cc:2781] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:09.955576 13089 raft_consensus.cc:686] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:09.955708 13089 consensus_queue.cc:237] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:09.957294 11887 catalog_manager.cc:5484] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:10.259224 13177 raft_consensus.cc:482] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:10.259377 13177 raft_consensus.cc:504] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:10.259747 13177 leader_election.cc:288] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:10.260121 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a89230883347d4a64d96d321247012" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:10.260123 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a89230883347d4a64d96d321247012" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:10.260262 12430 raft_consensus.cc:2454] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:10.260274 12167 raft_consensus.cc:2454] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:10.260535 12232 leader_election.cc:302] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:10.260695 13177 raft_consensus.cc:2781] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:10.260757 13177 raft_consensus.cc:482] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:10.260811 13177 raft_consensus.cc:3037] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.261687 13177 raft_consensus.cc:504] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:10.261997 13177 leader_election.cc:288] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:10.262312 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a89230883347d4a64d96d321247012" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:10.262311 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6a89230883347d4a64d96d321247012" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:10.262420 12167 raft_consensus.cc:3037] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.262432 12430 raft_consensus.cc:3037] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.263200 12430 raft_consensus.cc:2454] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:10.263370 12167 raft_consensus.cc:2454] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:10.263597 12232 leader_election.cc:302] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:10.263767 13177 raft_consensus.cc:2781] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:10.263876 13177 raft_consensus.cc:686] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:10.264020 13177 consensus_queue.cc:237] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:10.265622 11887 catalog_manager.cc:5484] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
W20240503 14:00:10.392577 13330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:10.392812 13330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:10.395731 13330 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:00:10.409485 13199 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:10.409657 13089 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240503 14:00:10.419195 13163 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:10.423066 13352 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:10.487995 12429 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:10.488380 13177 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:10.492723 12167 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:10.493088 13177 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 101.252 ms
time per row: 0.0506262 ms
I20240503 14:00:10.541672 11889 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51640:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:00:10.547092 12016 tablet_service.cc:1467] Processing CreateTablet for tablet 79b21b1e5292488aad442060b417f3c1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:10.547214 12408 tablet_service.cc:1467] Processing CreateTablet for tablet 79b21b1e5292488aad442060b417f3c1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:10.547468 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b21b1e5292488aad442060b417f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:10.547573 12408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b21b1e5292488aad442060b417f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:10.549456 13391 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:10.550187 12278 tablet_service.cc:1467] Processing CreateTablet for tablet 355b5456887b4c5694badc9d53b1809a (DEFAULT_TABLE table=rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:10.550406 13391 tablet_bootstrap.cc:654] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:10.550485 12278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355b5456887b4c5694badc9d53b1809a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:10.550436 12148 tablet_service.cc:1467] Processing CreateTablet for tablet d837845c6f644bca83315c9f7c45246d (DEFAULT_TABLE table=rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:10.550701 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d837845c6f644bca83315c9f7c45246d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:10.551035 13390 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:10.551496 13391 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:10.551581 13391 ts_tablet_manager.cc:1367] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:10.551992 13390 tablet_bootstrap.cc:654] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:10.552044 13391 raft_consensus.cc:348] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.552146 13391 raft_consensus.cc:374] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:10.552175 13391 raft_consensus.cc:729] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:10.552300 13391 consensus_queue.cc:260] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.553061 12277 tablet_service.cc:1467] Processing CreateTablet for tablet d837845c6f644bca83315c9f7c45246d (DEFAULT_TABLE table=rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:10.553292 12277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d837845c6f644bca83315c9f7c45246d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:10.552697 13391 ts_tablet_manager.cc:1398] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:10.553529 12016 tablet_service.cc:1467] Processing CreateTablet for tablet 355b5456887b4c5694badc9d53b1809a (DEFAULT_TABLE table=rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:10.553592 13390 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:10.553655 13390 ts_tablet_manager.cc:1367] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:10.553761 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355b5456887b4c5694badc9d53b1809a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:10.554069 13390 raft_consensus.cc:348] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.554157 13390 raft_consensus.cc:374] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:10.554183 13390 raft_consensus.cc:729] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:10.554296 13390 consensus_queue.cc:260] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.554589 13390 ts_tablet_manager.cc:1398] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:10.555030 12015 tablet_service.cc:1467] Processing CreateTablet for tablet d837845c6f644bca83315c9f7c45246d (DEFAULT_TABLE table=rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:10.550187 12279 tablet_service.cc:1467] Processing CreateTablet for tablet 79b21b1e5292488aad442060b417f3c1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:10.555227 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d837845c6f644bca83315c9f7c45246d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:10.555573 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b21b1e5292488aad442060b417f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:10.555861 13390 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:10.556748 13390 tablet_bootstrap.cc:654] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:10.557704 12408 tablet_service.cc:1467] Processing CreateTablet for tablet 355b5456887b4c5694badc9d53b1809a (DEFAULT_TABLE table=rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:10.557808 13393 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:10.557952 12408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355b5456887b4c5694badc9d53b1809a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:10.558604 13393 tablet_bootstrap.cc:654] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:10.559809 13390 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:10.559885 13390 ts_tablet_manager.cc:1367] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:00:10.560178 13393 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:10.560261 13393 ts_tablet_manager.cc:1367] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:10.560462 13391 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:10.560472 13390 raft_consensus.cc:348] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.560580 13390 raft_consensus.cc:374] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:10.560613 13390 raft_consensus.cc:729] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:10.560755 13394 tablet_bootstrap.cc:492] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:10.560737 13390 consensus_queue.cc:260] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.560729 13393 raft_consensus.cc:348] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.560832 13393 raft_consensus.cc:374] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:10.560869 13393 raft_consensus.cc:729] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:10.560992 13390 ts_tablet_manager.cc:1398] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:10.561012 13393 consensus_queue.cc:260] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.561141 13390 tablet_bootstrap.cc:492] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:10.561347 13391 tablet_bootstrap.cc:654] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:10.561383 13393 ts_tablet_manager.cc:1398] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:10.561548 13393 tablet_bootstrap.cc:492] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:10.561748 13394 tablet_bootstrap.cc:654] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:10.562041 13390 tablet_bootstrap.cc:654] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:10.562454 13393 tablet_bootstrap.cc:654] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:10.563050 13390 tablet_bootstrap.cc:492] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:10.563120 13390 ts_tablet_manager.cc:1367] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:10.563565 13390 raft_consensus.cc:348] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:10.563649 13390 raft_consensus.cc:374] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:10.563689 13390 raft_consensus.cc:729] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:10.563813 13390 consensus_queue.cc:260] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:10.563961 13393 tablet_bootstrap.cc:492] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:10.564047 13393 ts_tablet_manager.cc:1367] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:00:10.564077 13390 ts_tablet_manager.cc:1398] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:10.564618 13391 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:10.564705 13391 ts_tablet_manager.cc:1367] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:00:10.564687 13393 raft_consensus.cc:348] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:10.564776 13393 raft_consensus.cc:374] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:10.564813 13393 raft_consensus.cc:729] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:10.564947 13393 consensus_queue.cc:260] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:10.565147 13391 raft_consensus.cc:348] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.565243 13391 raft_consensus.cc:374] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:10.565277 13391 raft_consensus.cc:729] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:10.565383 13391 consensus_queue.cc:260] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.565646 13391 ts_tablet_manager.cc:1398] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:10.565855 13393 ts_tablet_manager.cc:1398] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:10.566007 13393 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:10.566871 13393 tablet_bootstrap.cc:654] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:10.568455 13394 tablet_bootstrap.cc:492] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:10.568569 13394 ts_tablet_manager.cc:1367] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:00:10.569275 13393 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:10.569296 13394 raft_consensus.cc:348] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:10.569414 13394 raft_consensus.cc:374] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:10.569466 13394 raft_consensus.cc:729] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:10.569587 13393 ts_tablet_manager.cc:1367] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:00:10.569799 13394 consensus_queue.cc:260] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:10.570153 13393 raft_consensus.cc:348] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.570231 13394 ts_tablet_manager.cc:1398] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:10.570324 13393 raft_consensus.cc:374] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:10.570391 13393 raft_consensus.cc:729] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:10.570515 13393 consensus_queue.cc:260] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.571151 13393 ts_tablet_manager.cc:1398] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:10.647046 13262 raft_consensus.cc:482] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:10.647195 13262 raft_consensus.cc:504] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.647650 13262 leader_election.cc:288] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:10.648015 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b21b1e5292488aad442060b417f3c1" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:10.648092 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b21b1e5292488aad442060b417f3c1" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:10.648146 12037 raft_consensus.cc:2454] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:10.648253 12429 raft_consensus.cc:2454] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:10.648492 12235 leader_election.cc:302] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:10.648669 13262 raft_consensus.cc:2781] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:10.648746 13262 raft_consensus.cc:482] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:10.648787 13262 raft_consensus.cc:3037] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.649574 13262 raft_consensus.cc:504] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.649857 13262 leader_election.cc:288] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Requested vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:10.650161 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b21b1e5292488aad442060b417f3c1" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:10.650185 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b21b1e5292488aad442060b417f3c1" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:10.650267 12037 raft_consensus.cc:3037] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.650278 12429 raft_consensus.cc:3037] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.650997 12429 raft_consensus.cc:2454] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:10.651046 12037 raft_consensus.cc:2454] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:10.651275 12232 leader_election.cc:302] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:10.651433 13262 raft_consensus.cc:2781] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:10.651515 13262 raft_consensus.cc:686] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:10.651656 13262 consensus_queue.cc:237] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.653200 11889 catalog_manager.cc:5484] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:10.663465 13089 raft_consensus.cc:482] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:10.663581 13089 raft_consensus.cc:504] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:10.663918 13089 leader_election.cc:288] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:10.664276 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d837845c6f644bca83315c9f7c45246d" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:10.664285 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d837845c6f644bca83315c9f7c45246d" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:10.664407 12298 raft_consensus.cc:2454] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:10.664404 12167 raft_consensus.cc:2454] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:10.664662 11973 leader_election.cc:302] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:10.664827 13089 raft_consensus.cc:2781] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:10.664882 13089 raft_consensus.cc:482] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:10.664932 13089 raft_consensus.cc:3037] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.665685 13089 raft_consensus.cc:504] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:10.665963 13089 leader_election.cc:288] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:10.666244 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d837845c6f644bca83315c9f7c45246d" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:10.666328 12167 raft_consensus.cc:3037] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.666299 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d837845c6f644bca83315c9f7c45246d" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:10.666400 12298 raft_consensus.cc:3037] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.667057 12167 raft_consensus.cc:2454] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:10.667356 11973 leader_election.cc:302] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:10.667510 13089 raft_consensus.cc:2781] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:10.667600 13089 raft_consensus.cc:686] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:10.667726 13089 consensus_queue.cc:237] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:10.669222 11889 catalog_manager.cc:5484] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:10.682211 13089 raft_consensus.cc:482] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:10.682300 13089 raft_consensus.cc:504] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.682636 13089 leader_election.cc:288] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:10.682986 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355b5456887b4c5694badc9d53b1809a" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:10.683018 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355b5456887b4c5694badc9d53b1809a" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:10.683115 12429 raft_consensus.cc:2454] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:10.683125 12296 raft_consensus.cc:2454] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:10.683363 11973 leader_election.cc:302] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:10.683498 13089 raft_consensus.cc:2781] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:10.683558 13089 raft_consensus.cc:482] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:10.683598 13089 raft_consensus.cc:3037] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.692095 12298 raft_consensus.cc:2454] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:10.708269 13089 raft_consensus.cc:504] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.708703 13089 leader_election.cc:288] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:10.709028 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355b5456887b4c5694badc9d53b1809a" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:10.709053 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355b5456887b4c5694badc9d53b1809a" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:10.709151 12429 raft_consensus.cc:3037] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.709159 12298 raft_consensus.cc:3037] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:10.709849 12429 raft_consensus.cc:2454] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:10.709874 12298 raft_consensus.cc:2454] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:10.710114 11973 leader_election.cc:302] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:10.710278 13089 raft_consensus.cc:2781] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:10.710390 13089 raft_consensus.cc:686] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:10.710522 13089 consensus_queue.cc:237] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:10.712319 11889 catalog_manager.cc:5484] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
W20240503 14:00:10.846609 13406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:10.846906 13406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:10.850117 13406 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 14:00:10.927393 12167 raft_consensus.cc:1261] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:10.927982 13089 consensus_queue.cc:1035] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:10.928040 12296 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:10.928095 12429 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:10.928393 12298 raft_consensus.cc:1261] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:10.929049 12429 raft_consensus.cc:1261] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:10.929817 13177 consensus_queue.cc:1035] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:10.930191 12037 raft_consensus.cc:1261] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:10.930419 13089 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:10.931660 13199 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:10.931932 13199 consensus_queue.cc:1035] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:10.936215 13262 consensus_queue.cc:1035] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.961 ms
time per row: 0.0519803 ms
I20240503 14:00:10.994899 11892 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51640:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:00:10.999665 12148 tablet_service.cc:1467] Processing CreateTablet for tablet a4b4e16598b740cb9d690c2442a3e994 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.000000 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b4e16598b740cb9d690c2442a3e994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:10.999941 12015 tablet_service.cc:1467] Processing CreateTablet for tablet a4b4e16598b740cb9d690c2442a3e994 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.000216 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b4e16598b740cb9d690c2442a3e994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.000202 12279 tablet_service.cc:1467] Processing CreateTablet for tablet c8933689df2c45b59a9cd16024ddb7d9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.000465 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8933689df2c45b59a9cd16024ddb7d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.000525 12408 tablet_service.cc:1467] Processing CreateTablet for tablet a4b4e16598b740cb9d690c2442a3e994 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.000779 12408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b4e16598b740cb9d690c2442a3e994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.002306 13390 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:11.002390 13394 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:11.002439 13393 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:11.003261 13390 tablet_bootstrap.cc:654] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.003254 13391 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:11.003365 13394 tablet_bootstrap.cc:654] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.003368 12410 tablet_service.cc:1467] Processing CreateTablet for tablet 4224ca4d93ce4784b52a7550468ce1b1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.003603 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4224ca4d93ce4784b52a7550468ce1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.004444 12148 tablet_service.cc:1467] Processing CreateTablet for tablet c8933689df2c45b59a9cd16024ddb7d9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.004678 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8933689df2c45b59a9cd16024ddb7d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.004681 13390 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:11.004760 13390 ts_tablet_manager.cc:1367] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:00:11.005203 13390 raft_consensus.cc:348] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.005307 13390 raft_consensus.cc:374] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.005338 13390 raft_consensus.cc:729] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.005436 12146 tablet_service.cc:1467] Processing CreateTablet for tablet 4224ca4d93ce4784b52a7550468ce1b1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.005463 13390 consensus_queue.cc:260] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.005642 12146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4224ca4d93ce4784b52a7550468ce1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.005741 13390 ts_tablet_manager.cc:1398] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:11.006008 13393 tablet_bootstrap.cc:654] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.006449 12408 tablet_service.cc:1467] Processing CreateTablet for tablet c8933689df2c45b59a9cd16024ddb7d9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.006698 12408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8933689df2c45b59a9cd16024ddb7d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.006898 12015 tablet_service.cc:1467] Processing CreateTablet for tablet 4224ca4d93ce4784b52a7550468ce1b1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.007040 13393 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:11.007112 13393 ts_tablet_manager.cc:1367] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 14:00:11.007133 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4224ca4d93ce4784b52a7550468ce1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.007656 13393 raft_consensus.cc:348] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.007766 13393 raft_consensus.cc:374] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.007813 13393 raft_consensus.cc:729] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.007958 13393 consensus_queue.cc:260] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.008232 13393 ts_tablet_manager.cc:1398] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:11.008924 13390 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:11.010329 13390 tablet_bootstrap.cc:654] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.011456 13390 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:11.011543 13390 ts_tablet_manager.cc:1367] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:00:11.011893 13390 raft_consensus.cc:348] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:11.011971 13390 raft_consensus.cc:374] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.012006 13390 raft_consensus.cc:729] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.012116 13390 consensus_queue.cc:260] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:11.012322 13390 ts_tablet_manager.cc:1398] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:11.027405 13391 tablet_bootstrap.cc:654] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.027770 13394 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:11.027887 13394 ts_tablet_manager.cc:1367] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.002s
I20240503 14:00:11.028398 13394 raft_consensus.cc:348] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.028512 13394 raft_consensus.cc:374] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.028543 13394 raft_consensus.cc:729] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.028661 13394 consensus_queue.cc:260] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.028717 13391 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:11.028805 13391 ts_tablet_manager.cc:1367] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 14:00:11.028959 13394 ts_tablet_manager.cc:1398] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:11.029095 13394 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:11.029275 13391 raft_consensus.cc:348] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.029395 13391 raft_consensus.cc:374] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.029435 13391 raft_consensus.cc:729] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.029572 13391 consensus_queue.cc:260] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.029891 13391 ts_tablet_manager.cc:1398] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:11.030006 13394 tablet_bootstrap.cc:654] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.030066 13391 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:11.030977 13391 tablet_bootstrap.cc:654] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.031826 13394 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:11.031917 13394 ts_tablet_manager.cc:1367] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:00:11.032004 13391 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:11.032068 13391 ts_tablet_manager.cc:1367] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:11.032353 13394 raft_consensus.cc:348] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.032461 13394 raft_consensus.cc:374] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.032490 13394 raft_consensus.cc:729] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.032588 13394 consensus_queue.cc:260] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.032745 13391 raft_consensus.cc:348] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:11.032824 13391 raft_consensus.cc:374] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.032836 13394 ts_tablet_manager.cc:1398] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:11.032856 13391 raft_consensus.cc:729] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.032999 13394 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:11.032959 13391 consensus_queue.cc:260] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:11.033241 13391 ts_tablet_manager.cc:1398] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:11.033387 13391 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:11.033875 13394 tablet_bootstrap.cc:654] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.034307 13391 tablet_bootstrap.cc:654] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.034905 13394 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:11.034992 13394 ts_tablet_manager.cc:1367] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:00:11.035409 13394 raft_consensus.cc:348] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:11.035488 13394 raft_consensus.cc:374] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.035521 13394 raft_consensus.cc:729] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.035568 13391 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:11.035645 13394 consensus_queue.cc:260] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:11.035760 13391 ts_tablet_manager.cc:1367] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:11.036182 13391 raft_consensus.cc:348] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.036240 13394 ts_tablet_manager.cc:1398] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:11.036278 13391 raft_consensus.cc:374] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.036320 13391 raft_consensus.cc:729] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.036444 13391 consensus_queue.cc:260] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.036736 13391 ts_tablet_manager.cc:1398] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:11.104036 13435 raft_consensus.cc:482] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:11.104158 13435 raft_consensus.cc:504] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:11.104593 13435 leader_election.cc:288] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:11.105017 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4224ca4d93ce4784b52a7550468ce1b1" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:11.105063 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4224ca4d93ce4784b52a7550468ce1b1" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:11.105176 12037 raft_consensus.cc:2454] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:11.105185 12167 raft_consensus.cc:2454] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:11.105458 12366 leader_election.cc:302] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:11.105604 13435 raft_consensus.cc:2781] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:11.105657 13435 raft_consensus.cc:482] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:11.105703 13435 raft_consensus.cc:3037] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.106575 13435 raft_consensus.cc:504] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:11.106871 13435 leader_election.cc:288] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Requested vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:11.107182 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4224ca4d93ce4784b52a7550468ce1b1" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:11.107228 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4224ca4d93ce4784b52a7550468ce1b1" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:11.107329 12037 raft_consensus.cc:3037] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.107403 12167 raft_consensus.cc:3037] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.108300 12037 raft_consensus.cc:2454] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 1.
I20240503 14:00:11.108300 12167 raft_consensus.cc:2454] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 1.
I20240503 14:00:11.108606 12366 leader_election.cc:302] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:11.108780 13435 raft_consensus.cc:2781] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:11.108879 13435 raft_consensus.cc:686] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Becoming Leader. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Running, Role: LEADER
I20240503 14:00:11.109006 13435 consensus_queue.cc:237] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:11.110589 11892 catalog_manager.cc:5484] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84c63c6f551949f6a1d7c07760824361 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:00:11.113176 13199 raft_consensus.cc:482] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:11.113256 13199 raft_consensus.cc:504] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.113548 13199 leader_election.cc:288] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:11.113829 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b4e16598b740cb9d690c2442a3e994" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:11.113855 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b4e16598b740cb9d690c2442a3e994" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:11.113940 12167 raft_consensus.cc:2454] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:11.113956 12430 raft_consensus.cc:2454] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:11.114203 11973 leader_election.cc:302] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:11.114356 13199 raft_consensus.cc:2781] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:11.114425 13199 raft_consensus.cc:482] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:11.114470 13199 raft_consensus.cc:3037] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.115276 13199 raft_consensus.cc:504] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.115574 13199 leader_election.cc:288] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:11.115854 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b4e16598b740cb9d690c2442a3e994" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:11.115900 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b4e16598b740cb9d690c2442a3e994" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:11.115947 12167 raft_consensus.cc:3037] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.115996 12430 raft_consensus.cc:3037] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.116645 12167 raft_consensus.cc:2454] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:11.116739 12430 raft_consensus.cc:2454] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:11.116904 11973 leader_election.cc:302] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:11.117098 13199 raft_consensus.cc:2781] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:11.117183 13199 raft_consensus.cc:686] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:11.117313 13199 consensus_queue.cc:237] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.118902 11892 catalog_manager.cc:5484] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:11.353998 13435 raft_consensus.cc:482] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:11.354175 13435 raft_consensus.cc:504] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.354555 13435 leader_election.cc:288] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:11.355001 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8933689df2c45b59a9cd16024ddb7d9" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:11.355057 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8933689df2c45b59a9cd16024ddb7d9" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:11.355141 12167 raft_consensus.cc:2454] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:11.355283 12298 raft_consensus.cc:2454] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:11.355497 12366 leader_election.cc:302] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:11.355669 13435 raft_consensus.cc:2781] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:11.355746 13435 raft_consensus.cc:482] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:11.355793 13435 raft_consensus.cc:3037] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.356653 13435 raft_consensus.cc:504] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.356951 13435 leader_election.cc:288] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:11.357276 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8933689df2c45b59a9cd16024ddb7d9" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:11.357328 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8933689df2c45b59a9cd16024ddb7d9" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:11.357391 12167 raft_consensus.cc:3037] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.357435 12298 raft_consensus.cc:3037] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.358412 12298 raft_consensus.cc:2454] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 1.
I20240503 14:00:11.358675 12366 leader_election.cc:302] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:11.358763 12167 raft_consensus.cc:2454] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 1.
I20240503 14:00:11.358848 13435 raft_consensus.cc:2781] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:11.358958 13435 raft_consensus.cc:686] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Becoming Leader. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Running, Role: LEADER
I20240503 14:00:11.359355 13435 consensus_queue.cc:237] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.361043 11892 catalog_manager.cc:5484] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84c63c6f551949f6a1d7c07760824361 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
W20240503 14:00:11.473100 13455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:11.473356 13455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:11.476464 13455 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 14:00:11.561828 12037 raft_consensus.cc:1261] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:11.562055 12168 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:11.562055 12167 raft_consensus.cc:1261] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:11.562255 12298 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:11.562539 13435 consensus_queue.cc:1035] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:11.562757 13486 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:11.562861 13145 consensus_queue.cc:1035] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:11.562883 13435 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:11.566789 12429 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:11.567628 13199 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:11.568419 12166 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:11.568802 13199 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:11.575089 13489 mvcc.cc:205] Tried to move back new op lower bound from 7023594748153229312 to 7023594747329376256. Current Snapshot: MvccSnapshot[applied={T|T < 7023594748153229312}]
I20240503 14:00:11.589017 13482 mvcc.cc:205] Tried to move back new op lower bound from 7023594748152934400 to 7023594746303594496. Current Snapshot: MvccSnapshot[applied={T|T < 7023594748152934400}]
I20240503 14:00:11.594823 13485 mvcc.cc:205] Tried to move back new op lower bound from 7023594748152934400 to 7023594746303594496. Current Snapshot: MvccSnapshot[applied={T|T < 7023594748152934400}]
INSERT report
rows total: 2000
time total: 119.124 ms
time per row: 0.0595621 ms
I20240503 14:00:11.637691 11892 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51640:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:00:11.643108 12146 tablet_service.cc:1467] Processing CreateTablet for tablet 2089eeda1e0f4ff5926584d472e6631d (DEFAULT_TABLE table=rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.643191 12015 tablet_service.cc:1467] Processing CreateTablet for tablet 2089eeda1e0f4ff5926584d472e6631d (DEFAULT_TABLE table=rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.643533 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2089eeda1e0f4ff5926584d472e6631d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.643533 12146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2089eeda1e0f4ff5926584d472e6631d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.643926 12408 tablet_service.cc:1467] Processing CreateTablet for tablet df13ec098dc548afac8c67195d25e7e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.644243 12408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df13ec098dc548afac8c67195d25e7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.644445 12279 tablet_service.cc:1467] Processing CreateTablet for tablet 2089eeda1e0f4ff5926584d472e6631d (DEFAULT_TABLE table=rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.644711 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2089eeda1e0f4ff5926584d472e6631d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.645990 13516 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:11.646071 13515 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:11.643925 12410 tablet_service.cc:1467] Processing CreateTablet for tablet 70ffc5888a5945d6a8e166420e6f1bd9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.646842 13516 tablet_bootstrap.cc:654] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.646842 13515 tablet_bootstrap.cc:654] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.646970 13518 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:11.647008 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ffc5888a5945d6a8e166420e6f1bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.647939 13515 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:11.647939 13516 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:11.647946 13518 tablet_bootstrap.cc:654] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.648032 13516 ts_tablet_manager.cc:1367] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:11.648032 13515 ts_tablet_manager.cc:1367] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:11.648473 13515 raft_consensus.cc:348] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.648473 13516 raft_consensus.cc:348] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.648591 13515 raft_consensus.cc:374] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.648591 13516 raft_consensus.cc:374] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.648634 13516 raft_consensus.cc:729] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.648634 13515 raft_consensus.cc:729] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.648756 13515 consensus_queue.cc:260] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.648756 13516 consensus_queue.cc:260] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.649034 13516 ts_tablet_manager.cc:1398] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:11.649037 13515 ts_tablet_manager.cc:1398] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:11.649861 12277 tablet_service.cc:1467] Processing CreateTablet for tablet 70ffc5888a5945d6a8e166420e6f1bd9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.649931 12015 tablet_service.cc:1467] Processing CreateTablet for tablet 70ffc5888a5945d6a8e166420e6f1bd9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.650113 12277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ffc5888a5945d6a8e166420e6f1bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.650120 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ffc5888a5945d6a8e166420e6f1bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.650137 12146 tablet_service.cc:1467] Processing CreateTablet for tablet df13ec098dc548afac8c67195d25e7e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.650358 12146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df13ec098dc548afac8c67195d25e7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.652334 13515 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:11.652514 12279 tablet_service.cc:1467] Processing CreateTablet for tablet df13ec098dc548afac8c67195d25e7e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:11.652729 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df13ec098dc548afac8c67195d25e7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:11.653105 13515 tablet_bootstrap.cc:654] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.653666 13516 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:11.654136 13515 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:11.654222 13515 ts_tablet_manager.cc:1367] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:11.654624 13516 tablet_bootstrap.cc:654] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.654808 13515 raft_consensus.cc:348] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:11.655031 13517 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:11.654917 13515 raft_consensus.cc:374] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.655117 13515 raft_consensus.cc:729] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.655298 13515 consensus_queue.cc:260] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:11.655622 13515 ts_tablet_manager.cc:1398] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:11.655769 13516 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:11.655836 13516 ts_tablet_manager.cc:1367] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:11.656152 13517 tablet_bootstrap.cc:654] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.656332 13516 raft_consensus.cc:348] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.656419 13516 raft_consensus.cc:374] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.656456 13516 raft_consensus.cc:729] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.656571 13516 consensus_queue.cc:260] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.656888 13516 ts_tablet_manager.cc:1398] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:11.657227 13517 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:11.657303 13517 ts_tablet_manager.cc:1367] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:11.657840 13517 raft_consensus.cc:348] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.657927 13517 raft_consensus.cc:374] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.657961 13517 raft_consensus.cc:729] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.658214 13517 consensus_queue.cc:260] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.658535 13517 ts_tablet_manager.cc:1398] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:11.659518 13517 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:11.659811 13518 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:11.659895 13518 ts_tablet_manager.cc:1367] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:00:11.660528 13517 tablet_bootstrap.cc:654] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.660477 13518 raft_consensus.cc:348] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.660581 13518 raft_consensus.cc:374] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.660614 13518 raft_consensus.cc:729] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.660748 13518 consensus_queue.cc:260] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.661072 13518 ts_tablet_manager.cc:1398] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:11.661284 13518 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:11.662068 13518 tablet_bootstrap.cc:654] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.662231 13517 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:11.662312 13517 ts_tablet_manager.cc:1367] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:11.662758 13517 raft_consensus.cc:348] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:11.662853 13517 raft_consensus.cc:374] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.662884 13517 raft_consensus.cc:729] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.663030 13518 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:11.663005 13517 consensus_queue.cc:260] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:11.663112 13518 ts_tablet_manager.cc:1367] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:11.663321 13517 ts_tablet_manager.cc:1398] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:11.663625 13518 raft_consensus.cc:348] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:11.663725 13518 raft_consensus.cc:374] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.663756 13518 raft_consensus.cc:729] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.663872 13518 consensus_queue.cc:260] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:11.664202 13518 ts_tablet_manager.cc:1398] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:11.664355 13518 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:11.664975 13518 tablet_bootstrap.cc:654] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:11.666003 13518 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:11.666080 13518 ts_tablet_manager.cc:1367] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:11.666458 13518 raft_consensus.cc:348] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.666559 13518 raft_consensus.cc:374] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:11.666604 13518 raft_consensus.cc:729] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:11.666735 13518 consensus_queue.cc:260] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.666976 13518 ts_tablet_manager.cc:1398] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:11.690636 13177 raft_consensus.cc:482] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:11.690764 13177 raft_consensus.cc:504] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.691129 13177 leader_election.cc:288] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:11.691504 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df13ec098dc548afac8c67195d25e7e8" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:11.691504 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df13ec098dc548afac8c67195d25e7e8" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:11.691648 12167 raft_consensus.cc:2454] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:11.691648 12429 raft_consensus.cc:2454] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:11.692242 12232 leader_election.cc:302] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:11.692397 13177 raft_consensus.cc:2781] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:11.692457 13177 raft_consensus.cc:482] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:11.692504 13177 raft_consensus.cc:3037] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.693289 13177 raft_consensus.cc:504] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.693585 13177 leader_election.cc:288] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Requested vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:11.693863 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df13ec098dc548afac8c67195d25e7e8" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:11.693884 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df13ec098dc548afac8c67195d25e7e8" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:11.693981 12167 raft_consensus.cc:3037] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.693992 12429 raft_consensus.cc:3037] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.694664 12167 raft_consensus.cc:2454] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:11.694778 12429 raft_consensus.cc:2454] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:11.694952 12235 leader_election.cc:302] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:11.695191 13177 raft_consensus.cc:2781] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:11.695391 13177 raft_consensus.cc:686] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:11.695534 13177 consensus_queue.cc:237] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:11.697153 11892 catalog_manager.cc:5484] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:11.755968 13376 raft_consensus.cc:482] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:11.756103 13376 raft_consensus.cc:504] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.756489 13376 leader_election.cc:288] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:11.756837 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2089eeda1e0f4ff5926584d472e6631d" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:11.756902 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2089eeda1e0f4ff5926584d472e6631d" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:11.756990 12167 raft_consensus.cc:2454] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:11.757041 12296 raft_consensus.cc:2454] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:11.757329 11973 leader_election.cc:302] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:11.757486 13376 raft_consensus.cc:2781] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:11.757550 13376 raft_consensus.cc:482] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:11.757586 13376 raft_consensus.cc:3037] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.758613 13376 raft_consensus.cc:504] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.758944 13376 leader_election.cc:288] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:11.759263 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2089eeda1e0f4ff5926584d472e6631d" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:11.759384 12167 raft_consensus.cc:3037] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.759380 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2089eeda1e0f4ff5926584d472e6631d" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:11.759483 12296 raft_consensus.cc:3037] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.760155 12167 raft_consensus.cc:2454] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:11.760408 12296 raft_consensus.cc:2454] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:11.760411 11973 leader_election.cc:302] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:11.760613 13376 raft_consensus.cc:2781] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:11.760699 13376 raft_consensus.cc:686] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:11.760838 13376 consensus_queue.cc:237] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:11.762557 11892 catalog_manager.cc:5484] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:11.795011 13487 raft_consensus.cc:482] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:11.795145 13487 raft_consensus.cc:504] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:11.795653 13487 leader_election.cc:288] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:11.795970 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ffc5888a5945d6a8e166420e6f1bd9" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:11.796105 12037 raft_consensus.cc:2454] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:11.796432 12366 leader_election.cc:302] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:11.796590 13487 raft_consensus.cc:2781] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:11.796556 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ffc5888a5945d6a8e166420e6f1bd9" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:11.796650 13487 raft_consensus.cc:482] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:11.796689 13487 raft_consensus.cc:3037] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.796720 12296 raft_consensus.cc:2454] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:11.797659 13487 raft_consensus.cc:504] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:11.797935 13487 leader_election.cc:288] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Requested vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:11.798281 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ffc5888a5945d6a8e166420e6f1bd9" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:11.798278 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ffc5888a5945d6a8e166420e6f1bd9" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:11.798380 12037 raft_consensus.cc:3037] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.798388 12296 raft_consensus.cc:3037] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:11.799130 12296 raft_consensus.cc:2454] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 1.
I20240503 14:00:11.799165 12037 raft_consensus.cc:2454] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 1.
I20240503 14:00:11.799402 12366 leader_election.cc:302] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:11.799573 13487 raft_consensus.cc:2781] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:11.799667 13487 raft_consensus.cc:686] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Becoming Leader. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Running, Role: LEADER
I20240503 14:00:11.799816 13487 consensus_queue.cc:237] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:11.801518 11890 catalog_manager.cc:5484] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84c63c6f551949f6a1d7c07760824361 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
W20240503 14:00:11.950314 13532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:11.950649 13532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:11.954759 13532 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 14:00:12.044077 12296 raft_consensus.cc:1261] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.044111 12037 raft_consensus.cc:1261] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.044828 12167 raft_consensus.cc:1261] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.044960 13486 consensus_queue.cc:1035] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:12.045109 12429 raft_consensus.cc:1261] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.045260 13486 consensus_queue.cc:1035] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:12.045573 13528 consensus_queue.cc:1035] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:12.047487 12167 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.047863 13199 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:12.051190 13177 consensus_queue.cc:1035] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:12.053164 12298 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.058759 13199 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.626 ms
time per row: 0.0558129 ms
I20240503 14:00:12.110388 11886 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51640:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:00:12.115959 12146 tablet_service.cc:1467] Processing CreateTablet for tablet dcdc3ae5232a49cdb003d9a2e3cf6d92 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.115995 12015 tablet_service.cc:1467] Processing CreateTablet for tablet dcdc3ae5232a49cdb003d9a2e3cf6d92 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.116294 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdc3ae5232a49cdb003d9a2e3cf6d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.116300 12146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdc3ae5232a49cdb003d9a2e3cf6d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.116503 12410 tablet_service.cc:1467] Processing CreateTablet for tablet dcdc3ae5232a49cdb003d9a2e3cf6d92 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.116781 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdc3ae5232a49cdb003d9a2e3cf6d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.118016 13515 tablet_bootstrap.cc:492] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:12.118496 13516 tablet_bootstrap.cc:492] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:12.119091 13515 tablet_bootstrap.cc:654] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.119416 13516 tablet_bootstrap.cc:654] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.119650 13517 tablet_bootstrap.cc:492] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:12.119650 12015 tablet_service.cc:1467] Processing CreateTablet for tablet b031f09b97664d1a94c14f0661181899 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.119705 12279 tablet_service.cc:1467] Processing CreateTablet for tablet b031f09b97664d1a94c14f0661181899 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.119870 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b031f09b97664d1a94c14f0661181899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.119989 12279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b031f09b97664d1a94c14f0661181899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.120724 13517 tablet_bootstrap.cc:654] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.120802 12146 tablet_service.cc:1467] Processing CreateTablet for tablet b031f09b97664d1a94c14f0661181899 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.121050 12146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b031f09b97664d1a94c14f0661181899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.122253 13515 tablet_bootstrap.cc:492] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:12.122279 12277 tablet_service.cc:1467] Processing CreateTablet for tablet 3481715d755a4277b0905b0832fad4af (DEFAULT_TABLE table=rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.122341 13515 ts_tablet_manager.cc:1367] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:00:12.122491 12277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3481715d755a4277b0905b0832fad4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.122838 13515 raft_consensus.cc:348] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.122946 13515 raft_consensus.cc:374] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.122978 13515 raft_consensus.cc:729] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.123087 13515 consensus_queue.cc:260] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.123313 13516 tablet_bootstrap.cc:492] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:12.123387 13516 ts_tablet_manager.cc:1367] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:00:12.123404 13515 ts_tablet_manager.cc:1398] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:12.123579 13515 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:12.123811 13516 raft_consensus.cc:348] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.123903 13516 raft_consensus.cc:374] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.123940 13516 raft_consensus.cc:729] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.124044 13516 consensus_queue.cc:260] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.124316 13516 ts_tablet_manager.cc:1398] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:12.124365 13515 tablet_bootstrap.cc:654] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.124465 13516 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:12.125077 13517 tablet_bootstrap.cc:492] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:12.125169 13517 ts_tablet_manager.cc:1367] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:00:12.125226 13516 tablet_bootstrap.cc:654] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.125519 13517 raft_consensus.cc:348] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.125617 13517 raft_consensus.cc:374] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.125643 13517 raft_consensus.cc:729] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.125742 13517 consensus_queue.cc:260] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.126109 13517 ts_tablet_manager.cc:1398] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:12.126426 13515 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:12.126502 13515 ts_tablet_manager.cc:1367] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:00:12.126921 13515 raft_consensus.cc:348] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.127012 13515 raft_consensus.cc:374] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.127041 13515 raft_consensus.cc:729] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.127140 13515 consensus_queue.cc:260] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.127445 13515 ts_tablet_manager.cc:1398] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:12.127528 12148 tablet_service.cc:1467] Processing CreateTablet for tablet 3481715d755a4277b0905b0832fad4af (DEFAULT_TABLE table=rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.127713 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3481715d755a4277b0905b0832fad4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.129277 13518 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:12.130002 13518 tablet_bootstrap.cc:654] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.130910 13518 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:12.131006 13518 ts_tablet_manager.cc:1367] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:12.131431 13518 raft_consensus.cc:348] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.131520 13518 raft_consensus.cc:374] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.131559 13518 raft_consensus.cc:729] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.131677 13518 consensus_queue.cc:260] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.131949 13518 ts_tablet_manager.cc:1398] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:12.132110 13518 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:12.132767 12410 tablet_service.cc:1467] Processing CreateTablet for tablet 3481715d755a4277b0905b0832fad4af (DEFAULT_TABLE table=rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.133026 13518 tablet_bootstrap.cc:654] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.133024 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3481715d755a4277b0905b0832fad4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.134151 13518 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:12.134225 13518 ts_tablet_manager.cc:1367] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:12.134577 13518 raft_consensus.cc:348] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:12.134658 13518 raft_consensus.cc:374] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.134693 13518 raft_consensus.cc:729] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.134828 13518 consensus_queue.cc:260] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:12.134908 13517 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:12.135191 13518 ts_tablet_manager.cc:1398] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:12.135775 13517 tablet_bootstrap.cc:654] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.136680 13517 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:12.136770 13517 ts_tablet_manager.cc:1367] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:12.137156 13517 raft_consensus.cc:348] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:12.137233 13517 raft_consensus.cc:374] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.137254 13517 raft_consensus.cc:729] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.137351 13517 consensus_queue.cc:260] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:12.137569 13517 ts_tablet_manager.cc:1398] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:12.139899 13516 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:12.139992 13516 ts_tablet_manager.cc:1367] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 14:00:12.140353 13516 raft_consensus.cc:348] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.140447 13516 raft_consensus.cc:374] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.140473 13516 raft_consensus.cc:729] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.140578 13516 consensus_queue.cc:260] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.140843 13516 ts_tablet_manager.cc:1398] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:12.140993 13516 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:12.142791 13516 tablet_bootstrap.cc:654] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.147818 13516 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:12.147898 13516 ts_tablet_manager.cc:1367] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 14:00:12.148259 13516 raft_consensus.cc:348] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:12.148356 13516 raft_consensus.cc:374] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.148389 13516 raft_consensus.cc:729] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.148509 13516 consensus_queue.cc:260] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:12.148756 13516 ts_tablet_manager.cc:1398] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:12.159050 13528 raft_consensus.cc:482] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:12.159163 13528 raft_consensus.cc:504] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:12.159593 13528 leader_election.cc:288] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:12.159921 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3481715d755a4277b0905b0832fad4af" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:12.159940 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3481715d755a4277b0905b0832fad4af" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:12.160058 12167 raft_consensus.cc:2454] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:12.160065 12429 raft_consensus.cc:2454] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:12.160344 12232 leader_election.cc:302] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:12.160545 13528 raft_consensus.cc:2781] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:12.160596 13528 raft_consensus.cc:482] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:12.160646 13528 raft_consensus.cc:3037] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.161659 13528 raft_consensus.cc:504] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:12.161988 13528 leader_election.cc:288] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:12.162292 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3481715d755a4277b0905b0832fad4af" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:12.162312 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3481715d755a4277b0905b0832fad4af" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:12.162395 12167 raft_consensus.cc:3037] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.162411 12429 raft_consensus.cc:3037] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.163328 12429 raft_consensus.cc:2454] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:12.163353 12167 raft_consensus.cc:2454] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:12.163606 12232 leader_election.cc:302] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:12.163751 13528 raft_consensus.cc:2781] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:12.163834 13528 raft_consensus.cc:686] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:12.163969 13528 consensus_queue.cc:237] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:12.165542 11886 catalog_manager.cc:5484] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:12.174414 13528 raft_consensus.cc:482] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:12.174507 13528 raft_consensus.cc:504] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.174810 13528 leader_election.cc:288] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:12.175120 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b031f09b97664d1a94c14f0661181899" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:12.175268 12167 raft_consensus.cc:2454] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:12.175209 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b031f09b97664d1a94c14f0661181899" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:12.175372 12037 raft_consensus.cc:2454] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:12.175555 12235 leader_election.cc:302] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:12.175712 13528 raft_consensus.cc:2781] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:12.175763 13528 raft_consensus.cc:482] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:12.175825 13528 raft_consensus.cc:3037] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.176546 13528 raft_consensus.cc:504] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.176823 13528 leader_election.cc:288] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:12.177073 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b031f09b97664d1a94c14f0661181899" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:12.177160 12167 raft_consensus.cc:3037] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.177134 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b031f09b97664d1a94c14f0661181899" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:12.177240 12037 raft_consensus.cc:3037] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.177964 12037 raft_consensus.cc:2454] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:12.178088 12167 raft_consensus.cc:2454] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:12.178215 12235 leader_election.cc:302] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:12.178345 13528 raft_consensus.cc:2781] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:12.178436 13528 raft_consensus.cc:686] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:12.178563 13528 consensus_queue.cc:237] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.180123 11886 catalog_manager.cc:5484] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:12.228276 13199 raft_consensus.cc:482] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:12.228390 13199 raft_consensus.cc:504] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.228703 13199 leader_election.cc:288] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:12.229028 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdc3ae5232a49cdb003d9a2e3cf6d92" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:12.229033 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdc3ae5232a49cdb003d9a2e3cf6d92" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:12.229146 12168 raft_consensus.cc:2454] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:12.229162 12429 raft_consensus.cc:2454] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:12.229401 11973 leader_election.cc:302] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:12.229542 13199 raft_consensus.cc:2781] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:12.229601 13199 raft_consensus.cc:482] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:12.229635 13199 raft_consensus.cc:3037] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.230456 13199 raft_consensus.cc:504] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.230731 13199 leader_election.cc:288] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:12.231007 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdc3ae5232a49cdb003d9a2e3cf6d92" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:12.231040 12168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcdc3ae5232a49cdb003d9a2e3cf6d92" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:12.231106 12429 raft_consensus.cc:3037] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.231150 12168 raft_consensus.cc:3037] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.231894 12429 raft_consensus.cc:2454] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:12.231966 12168 raft_consensus.cc:2454] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:12.232165 11970 leader_election.cc:302] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:12.232342 13199 raft_consensus.cc:2781] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:12.232424 13199 raft_consensus.cc:686] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:12.232726 13199 consensus_queue.cc:237] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:12.234315 11886 catalog_manager.cc:5484] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
W20240503 14:00:12.354082 13581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:12.354348 13581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:12.357419 13581 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 14:00:12.430063 12037 raft_consensus.cc:1261] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.430181 12168 raft_consensus.cc:1261] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.430833 12429 raft_consensus.cc:1261] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.431035 12168 raft_consensus.cc:1261] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.431047 13177 consensus_queue.cc:1035] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:12.431352 13177 consensus_queue.cc:1035] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:12.431453 13608 consensus_queue.cc:1035] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:12.431636 13177 consensus_queue.cc:1035] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:12.434302 12430 raft_consensus.cc:1261] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.434872 13376 consensus_queue.cc:1035] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:12.437377 12166 raft_consensus.cc:1261] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.438997 13199 consensus_queue.cc:1035] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:12.458242 13493 mvcc.cc:205] Tried to move back new op lower bound from 7023594751708561408 to 7023594750684508160. Current Snapshot: MvccSnapshot[applied={T|T < 7023594751708561408}]
INSERT report
rows total: 2000
time total: 87.6171 ms
time per row: 0.0438086 ms
I20240503 14:00:12.492415 11893 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51640:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:00:12.497563 12277 tablet_service.cc:1467] Processing CreateTablet for tablet e92d8f55591346158b9583feb958a8ca (DEFAULT_TABLE table=rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.497767 12016 tablet_service.cc:1467] Processing CreateTablet for tablet 92506cde7b084b14b88d4a915731eeb3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.497915 12277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92d8f55591346158b9583feb958a8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.498068 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92506cde7b084b14b88d4a915731eeb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.498090 12015 tablet_service.cc:1467] Processing CreateTablet for tablet e92d8f55591346158b9583feb958a8ca (DEFAULT_TABLE table=rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.498297 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92d8f55591346158b9583feb958a8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.499795 13518 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:12.500238 13515 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:12.500591 13518 tablet_bootstrap.cc:654] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.500665 12410 tablet_service.cc:1467] Processing CreateTablet for tablet e92d8f55591346158b9583feb958a8ca (DEFAULT_TABLE table=rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.500725 12408 tablet_service.cc:1467] Processing CreateTablet for tablet 92506cde7b084b14b88d4a915731eeb3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.500973 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92d8f55591346158b9583feb958a8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.501036 13515 tablet_bootstrap.cc:654] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.501051 12409 tablet_service.cc:1467] Processing CreateTablet for tablet 0e442bb664f44887aa8afdb62e1d97fc (DEFAULT_TABLE table=rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.502102 13518 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:12.502187 13518 ts_tablet_manager.cc:1367] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:12.502429 12016 tablet_service.cc:1467] Processing CreateTablet for tablet 0e442bb664f44887aa8afdb62e1d97fc (DEFAULT_TABLE table=rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.502580 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e442bb664f44887aa8afdb62e1d97fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.502570 13518 raft_consensus.cc:348] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.502665 13518 raft_consensus.cc:374] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.502705 13518 raft_consensus.cc:729] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.502830 13518 consensus_queue.cc:260] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.503088 13518 ts_tablet_manager.cc:1398] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:12.503136 12148 tablet_service.cc:1467] Processing CreateTablet for tablet 0e442bb664f44887aa8afdb62e1d97fc (DEFAULT_TABLE table=rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.503425 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e442bb664f44887aa8afdb62e1d97fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.504796 13515 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:12.504873 13515 ts_tablet_manager.cc:1367] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 14:00:12.505187 13516 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:12.505280 13515 raft_consensus.cc:348] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.505391 13515 raft_consensus.cc:374] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.505426 13515 raft_consensus.cc:729] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.505532 13515 consensus_queue.cc:260] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.505806 13515 ts_tablet_manager.cc:1398] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:12.505980 13515 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:12.506063 13516 tablet_bootstrap.cc:654] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.507141 13515 tablet_bootstrap.cc:654] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.507318 12409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e442bb664f44887aa8afdb62e1d97fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.507501 12277 tablet_service.cc:1467] Processing CreateTablet for tablet 92506cde7b084b14b88d4a915731eeb3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:12.507745 12277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92506cde7b084b14b88d4a915731eeb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.507764 13516 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143: No bootstrap required, opened a new log
I20240503 14:00:12.507835 13516 ts_tablet_manager.cc:1367] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:12.508247 13516 raft_consensus.cc:348] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:12.508399 13516 raft_consensus.cc:374] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.508433 13516 raft_consensus.cc:729] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.508550 13516 consensus_queue.cc:260] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:12.508812 13516 ts_tablet_manager.cc:1398] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:12.509631 13517 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:12.510010 13518 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:12.510186 12408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92506cde7b084b14b88d4a915731eeb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:12.510500 13515 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:12.510535 13517 tablet_bootstrap.cc:654] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.510572 13515 ts_tablet_manager.cc:1367] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:00:12.510994 13518 tablet_bootstrap.cc:654] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.511032 13515 raft_consensus.cc:348] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.511135 13515 raft_consensus.cc:374] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.511173 13515 raft_consensus.cc:729] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.511304 13515 consensus_queue.cc:260] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.511569 13515 ts_tablet_manager.cc:1398] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:12.511760 13515 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:12.512614 13515 tablet_bootstrap.cc:654] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.512974 13517 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:12.513047 13517 ts_tablet_manager.cc:1367] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:12.513175 13518 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: No bootstrap required, opened a new log
I20240503 14:00:12.513249 13518 ts_tablet_manager.cc:1367] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:12.513473 13517 raft_consensus.cc:348] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:12.513566 13517 raft_consensus.cc:374] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.513594 13517 raft_consensus.cc:729] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.513666 13518 raft_consensus.cc:348] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.513767 13518 raft_consensus.cc:374] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.513716 13517 consensus_queue.cc:260] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:12.513798 13518 raft_consensus.cc:729] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.513911 13518 consensus_queue.cc:260] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.514624 13518 ts_tablet_manager.cc:1398] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:12.515586 13515 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f: No bootstrap required, opened a new log
I20240503 14:00:12.515780 13515 ts_tablet_manager.cc:1367] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:00:12.516784 13515 raft_consensus.cc:348] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:12.517078 13515 raft_consensus.cc:374] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.517153 13515 raft_consensus.cc:729] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.517356 13515 consensus_queue.cc:260] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:12.517843 13515 ts_tablet_manager.cc:1398] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:12.518955 13517 ts_tablet_manager.cc:1398] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 14:00:12.519171 13517 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:12.520022 13517 tablet_bootstrap.cc:654] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.521112 13517 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:12.521203 13517 ts_tablet_manager.cc:1367] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:00:12.521633 13517 raft_consensus.cc:348] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.521762 13517 raft_consensus.cc:374] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.521807 13517 raft_consensus.cc:729] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.521927 13517 consensus_queue.cc:260] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.522177 13517 ts_tablet_manager.cc:1398] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:12.522358 13517 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:12.523134 13517 tablet_bootstrap.cc:654] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:12.524134 13517 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: No bootstrap required, opened a new log
I20240503 14:00:12.524209 13517 ts_tablet_manager.cc:1367] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:12.524612 13517 raft_consensus.cc:348] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.524768 13517 raft_consensus.cc:374] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:12.524812 13517 raft_consensus.cc:729] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:12.524955 13517 consensus_queue.cc:260] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.525210 13517 ts_tablet_manager.cc:1398] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:12.560802 13199 raft_consensus.cc:482] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:12.560940 13199 raft_consensus.cc:504] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:12.561389 13199 leader_election.cc:288] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:12.561738 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e442bb664f44887aa8afdb62e1d97fc" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:12.561769 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e442bb664f44887aa8afdb62e1d97fc" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143" is_pre_election: true
I20240503 14:00:12.561898 12429 raft_consensus.cc:2454] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:12.561908 12167 raft_consensus.cc:2454] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 0.
I20240503 14:00:12.562161 11970 leader_election.cc:302] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:12.562312 13199 raft_consensus.cc:2781] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:12.562369 13199 raft_consensus.cc:482] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:12.562415 13199 raft_consensus.cc:3037] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.563315 13199 raft_consensus.cc:504] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:12.563616 13199 leader_election.cc:288] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Requested vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:12.563941 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e442bb664f44887aa8afdb62e1d97fc" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:12.563941 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e442bb664f44887aa8afdb62e1d97fc" candidate_uuid: "623e8dcb58544e558207d8e652dfed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:12.564059 12167 raft_consensus.cc:3037] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.564070 12429 raft_consensus.cc:3037] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.564986 12429 raft_consensus.cc:2454] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:12.564986 12167 raft_consensus.cc:2454] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623e8dcb58544e558207d8e652dfed5f in term 1.
I20240503 14:00:12.565296 11970 leader_election.cc:302] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:12.565493 13199 raft_consensus.cc:2781] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:12.565584 13199 raft_consensus.cc:686] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Leader. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:12.565724 13199 consensus_queue.cc:237] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:12.567377 11894 catalog_manager.cc:5484] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 623e8dcb58544e558207d8e652dfed5f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:00:12.758513 13608 raft_consensus.cc:482] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:12.758652 13608 raft_consensus.cc:504] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.759044 13615 raft_consensus.cc:482] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:12.759086 13608 leader_election.cc:288] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:12.759160 13615 raft_consensus.cc:504] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.759512 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d8f55591346158b9583feb958a8ca" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:12.759617 13615 leader_election.cc:288] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:12.759656 12037 raft_consensus.cc:2454] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:12.759797 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d8f55591346158b9583feb958a8ca" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:12.759912 12429 raft_consensus.cc:2454] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:12.760003 12235 leader_election.cc:302] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:12.760227 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d8f55591346158b9583feb958a8ca" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:12.760231 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d8f55591346158b9583feb958a8ca" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee" is_pre_election: true
I20240503 14:00:12.760319 12037 raft_consensus.cc:2454] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:12.760334 12298 raft_consensus.cc:2454] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 0.
I20240503 14:00:12.760564 13608 raft_consensus.cc:2781] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:12.760551 12366 leader_election.cc:302] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:12.760624 13608 raft_consensus.cc:482] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:12.760668 13608 raft_consensus.cc:3037] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.760803 13615 raft_consensus.cc:2781] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:12.760874 13615 raft_consensus.cc:482] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:12.760915 13615 raft_consensus.cc:3037] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.761746 13608 raft_consensus.cc:504] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.761821 13615 raft_consensus.cc:504] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.762138 13615 leader_election.cc:288] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Requested vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:12.762264 13608 leader_election.cc:288] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Requested vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:12.762408 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d8f55591346158b9583feb958a8ca" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:12.762408 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d8f55591346158b9583feb958a8ca" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:12.762498 12037 raft_consensus.cc:3037] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.762555 12429 raft_consensus.cc:2379] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ca5386f84994496fa5376624391728ee in current term 1: Already voted for candidate 84c63c6f551949f6a1d7c07760824361 in this term.
I20240503 14:00:12.762828 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d8f55591346158b9583feb958a8ca" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:12.763401 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d8f55591346158b9583feb958a8ca" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:12.763532 12037 raft_consensus.cc:2454] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:12.763566 12298 raft_consensus.cc:2379] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 84c63c6f551949f6a1d7c07760824361 in current term 1: Already voted for candidate ca5386f84994496fa5376624391728ee in this term.
I20240503 14:00:12.763762 12235 leader_election.cc:302] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, ca5386f84994496fa5376624391728ee; no voters: 84c63c6f551949f6a1d7c07760824361
I20240503 14:00:12.763839 12366 leader_election.cc:302] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 623e8dcb58544e558207d8e652dfed5f, ca5386f84994496fa5376624391728ee
I20240503 14:00:12.763979 13608 raft_consensus.cc:2781] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:12.764020 13615 raft_consensus.cc:2726] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:00:12.764065 13608 raft_consensus.cc:686] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:12.764215 13608 consensus_queue.cc:237] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.765913 11894 catalog_manager.cc:5484] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:12.838069 13608 raft_consensus.cc:482] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:12.838238 13608 raft_consensus.cc:504] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.838651 13608 leader_election.cc:288] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:12.839092 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92506cde7b084b14b88d4a915731eeb3" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361" is_pre_election: true
I20240503 14:00:12.839257 12429 raft_consensus.cc:2454] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:12.839577 12232 leader_election.cc:302] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:12.839814 13608 raft_consensus.cc:2781] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:12.839850 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92506cde7b084b14b88d4a915731eeb3" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:12.839931 13608 raft_consensus.cc:482] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:12.840014 12037 raft_consensus.cc:2454] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 0.
I20240503 14:00:12.840081 13608 raft_consensus.cc:3037] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.841251 13608 raft_consensus.cc:504] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.841655 13608 leader_election.cc:288] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Requested vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:12.841924 12430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92506cde7b084b14b88d4a915731eeb3" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:12.841933 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92506cde7b084b14b88d4a915731eeb3" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:12.842031 12037 raft_consensus.cc:3037] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.842041 12430 raft_consensus.cc:3037] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:12.842773 12430 raft_consensus.cc:2454] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:12.842819 12037 raft_consensus.cc:2454] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 1.
I20240503 14:00:12.843124 12235 leader_election.cc:302] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:12.843297 13608 raft_consensus.cc:2781] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:12.843376 13608 raft_consensus.cc:686] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:12.843513 13608 consensus_queue.cc:237] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:12.845119 11894 catalog_manager.cc:5484] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
W20240503 14:00:12.990442 13630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:12.990682 13630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:12.993840 13630 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 14:00:13.033722 13199 consensus_queue.cc:1035] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:13.041553 13376 consensus_queue.cc:1035] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:13.081758 12037 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:13.083446 13608 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:13.086342 12430 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:13.092376 13528 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:13.093712 12036 raft_consensus.cc:1261] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:13.094955 12429 raft_consensus.cc:1261] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:13.095263 13608 consensus_queue.cc:1035] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:13.099505 13608 consensus_queue.cc:1035] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 101.259 ms
time per row: 0.0506293 ms
I20240503 14:00:13.148661 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:37005
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:44677
--webserver_port=46713
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:00:13.256686 13693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:13.256903 13693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:13.256964 13693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:13.259842 13693 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:13.259909 13693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:13.259990 13693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 14:00:13.263515 13693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:37005
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=46713
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.13693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:13.264403 13693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:13.265514 13693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:13.274331 13701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.275478 13699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.276351 13698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:13.277344 13693 server_base.cc:1034] running on GCE node
I20240503 14:00:13.277730 13693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:13.278375 13693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:13.279520 13693 hybrid_clock.cc:648] HybridClock initialized: now 1714744813279506 us; error 32 us; skew 500 ppm
I20240503 14:00:13.281379 13693 webserver.cc:460] Webserver started at http://127.0.99.133:46713/ using document root <none> and password file <none>
I20240503 14:00:13.281905 13693 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:13.281973 13693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:13.285254 13693 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:00:13.287029 13707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.288014 13693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:13.288149 13693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "66190085d1c04e068d5c0dec9c4bf07d"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:13.288599 13693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:13.329288 13693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:13.329876 13693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:13.330008 13693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:13.330477 13693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:13.331336 13693 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:13.331383 13693 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.331431 13693 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:13.331452 13693 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.340301 13693 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:37005
I20240503 14:00:13.340333 13820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:37005 every 8 connection(s)
I20240503 14:00:13.341404 13693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:00:13.345400 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 13693
I20240503 14:00:13.345875 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:37075
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:44677
--webserver_port=34509
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:00:13.352775 13821 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:13.352995 13821 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:13.353413 13821 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:13.354445 11894 ts_manager.cc:194] Re-registered known tserver with Master: 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005)
I20240503 14:00:13.354988 11894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:35735
W20240503 14:00:13.457844 13824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:13.458091 13824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:13.458159 13824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:13.461153 13824 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:13.461223 13824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:13.461308 13824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 14:00:13.464921 13824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:37075
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=34509
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.13824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:13.465759 13824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:13.466812 13824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:13.475528 13832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.476655 13830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.477497 13829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:13.478381 13824 server_base.cc:1034] running on GCE node
I20240503 14:00:13.478793 13824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:13.479357 13824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:13.480502 13824 hybrid_clock.cc:648] HybridClock initialized: now 1714744813480489 us; error 29 us; skew 500 ppm
I20240503 14:00:13.482863 13824 webserver.cc:460] Webserver started at http://127.0.99.134:34509/ using document root <none> and password file <none>
I20240503 14:00:13.483453 13824 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:13.483534 13824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:13.486779 13824 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:00:13.488744 13838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.489800 13824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:13.489909 13824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "b67f337e10994234baedbf0b19c13513"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:13.490372 13824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:13.505867 13824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:13.506394 13824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:13.506534 13824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:13.507105 13824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:13.508000 13824 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:13.508049 13824 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.508105 13824 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:13.508131 13824 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.516425 13824 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:37075
I20240503 14:00:13.516449 13951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:37075 every 8 connection(s)
I20240503 14:00:13.517428 13824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:00:13.522719 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 13824
I20240503 14:00:13.523080 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:46845
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:44677
--webserver_port=39327
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:00:13.525938 13952 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:13.526178 13952 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:13.526701 13952 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:13.527734 11894 ts_manager.cc:194] Re-registered known tserver with Master: b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075)
I20240503 14:00:13.528259 11894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:38847
W20240503 14:00:13.634760 13955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:13.634984 13955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:13.635047 13955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:13.638105 13955 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:13.638177 13955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:13.638263 13955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 14:00:13.642077 13955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:46845
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=39327
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.13955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:13.643003 13955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:13.644133 13955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:13.652921 13960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.654147 13961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.654918 13963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:13.655799 13955 server_base.cc:1034] running on GCE node
I20240503 14:00:13.656200 13955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:13.656710 13955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:13.657840 13955 hybrid_clock.cc:648] HybridClock initialized: now 1714744813657822 us; error 33 us; skew 500 ppm
I20240503 14:00:13.659715 13955 webserver.cc:460] Webserver started at http://127.0.99.135:39327/ using document root <none> and password file <none>
I20240503 14:00:13.660238 13955 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:13.660310 13955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:13.663555 13955 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:00:13.665479 13969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.666424 13955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:13.666532 13955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:13.666921 13955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:13.680963 13955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:13.681502 13955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:13.681648 13955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:13.682134 13955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:13.682968 13955 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:13.683014 13955 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.683074 13955 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:13.683102 13955 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.691756 13955 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:46845
I20240503 14:00:13.691774 14082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:46845 every 8 connection(s)
I20240503 14:00:13.692802 13955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:00:13.699563 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 13955
I20240503 14:00:13.699970 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
/tmp/dist-test-taskGjG1E4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:34763
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:44677
--webserver_port=40097
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:00:13.701429 14083 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:13.701670 14083 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:13.702085 14083 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:13.702965 11894 ts_manager.cc:194] Re-registered known tserver with Master: e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
I20240503 14:00:13.703510 11894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:36225
W20240503 14:00:13.819507 14086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:13.819725 14086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:00:13.819788 14086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:13.822702 14086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:00:13.822772 14086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:00:13.822863 14086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 14:00:13.826269 14086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:34763
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=40097
--tserver_master_addrs=127.0.99.190:44677
--never_fsync=true
--heap_profile_path=/tmp/kudu.14086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 13:43:16 UTC on 1a15035a6830
build id 105
I20240503 14:00:13.827162 14086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:00:13.828330 14086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:13.837548 14092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.838328 14091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:13.839349 14094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:13.840337 14086 server_base.cc:1034] running on GCE node
I20240503 14:00:13.840685 14086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:00:13.841219 14086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:00:13.842376 14086 hybrid_clock.cc:648] HybridClock initialized: now 1714744813842353 us; error 35 us; skew 500 ppm
I20240503 14:00:13.844527 14086 webserver.cc:460] Webserver started at http://127.0.99.136:40097/ using document root <none> and password file <none>
I20240503 14:00:13.845088 14086 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:13.845170 14086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:13.848639 14086 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:00:13.850683 14100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.851936 14086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:13.852054 14086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1b4a61e8a11440e7a9f4b300d2442f75"
format_stamp: "Formatted at 2024-05-03 14:00:03 on dist-test-slave-snql"
I20240503 14:00:13.852478 14086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:13.868899 14086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:13.869441 14086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:00:13.869575 14086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:13.870114 14086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:13.870985 14086 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:13.871038 14086 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.871091 14086 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:13.871114 14086 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:13.879101 14086 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:34763
I20240503 14:00:13.879139 14213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:34763 every 8 connection(s)
I20240503 14:00:13.880175 14086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:00:13.887748 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu as pid 14086
I20240503 14:00:13.888420 14214 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44677
I20240503 14:00:13.888650 14214 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:13.889135 14214 heartbeater.cc:511] Master 127.0.99.190:44677 requested a full tablet report, sending...
I20240503 14:00:13.890183 11894 ts_manager.cc:194] Re-registered known tserver with Master: 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763)
I20240503 14:00:13.890903 11894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:44905
I20240503 14:00:14.241849 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:14.244319 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.252501 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.255945 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:14.264243 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:14.277915 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.287771 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.289427 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.290351 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.292117 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:14.300302 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:14.335685 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:14.335795 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:14.336555 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.344139 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.344712 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:14.356956 13821 heartbeater.cc:503] Master 127.0.99.190:44677 was elected leader, sending a full tablet report...
I20240503 14:00:14.429504 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:14.454002 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.454790 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.476994 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.479993 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.483661 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:14.488306 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:14.490756 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.512507 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:14.513963 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:14.530563 13952 heartbeater.cc:503] Master 127.0.99.190:44677 was elected leader, sending a full tablet report...
I20240503 14:00:14.534592 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:14.535825 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.543756 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:14.550792 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:14.551502 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.580000 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.579993 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.592427 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.593269 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:14.597759 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:14.601935 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:14.609109 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.628399 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:14.640511 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.705780 14083 heartbeater.cc:503] Master 127.0.99.190:44677 was elected leader, sending a full tablet report...
I20240503 14:00:14.760764 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:14.797295 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:14.798635 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:14.800478 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.824877 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.826434 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.845564 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.846158 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:14.884220 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:14.885181 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:14.892243 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.893277 14214 heartbeater.cc:503] Master 127.0.99.190:44677 was elected leader, sending a full tablet report...
I20240503 14:00:14.907366 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.941421 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:14.952283 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:14.954471 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:14.956190 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.971755 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.983640 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:14.988152 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.992377 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:14.997376 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:15.003448 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.015024 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:15.016623 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:15.029734 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:15.031886 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.045177 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.047329 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.061378 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:15.062597 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:15.062816 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.071568 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:15.071988 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.072544 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.084718 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.091490 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.093123 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:15.121024 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.167311 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:15.169466 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:15.374392 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:15.376060 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.383816 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:15.384678 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.403651 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:15.412743 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:15.414897 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.416021 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.425446 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.442768 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.451906 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:15.452471 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:15.471873 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:15.478588 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.490526 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.493651 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:15.494942 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:15.501992 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.509609 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.530611 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:15.537134 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.561671 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.592728 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:15.600781 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:15.650067 12298 consensus_queue.cc:237] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:15.655499 12037 raft_consensus.cc:1261] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.656035 12428 raft_consensus.cc:1261] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.656579 14943 consensus_queue.cc:1035] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:15.660666 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1e9155230c74e92b6ee441de713a419. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:15.661954 13674 consensus_queue.cc:1035] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.676962 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:15.678844 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:15.680248 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.683833 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.689442 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:15.690486 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.691651 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:15.692893 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.693665 14943 raft_consensus.cc:2932] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.696331 12428 raft_consensus.cc:2932] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.699054 11890 catalog_manager.cc:5484] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.700114 12036 raft_consensus.cc:2932] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.706436 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.716599 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:15.717227 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:15.719712 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.724419 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:15.725067 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.730387 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:15.739707 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:15.811095 12296 consensus_queue.cc:237] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:15.814180 12036 raft_consensus.cc:1261] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.815126 12428 raft_consensus.cc:1261] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.815388 14943 consensus_queue.cc:1035] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.815719 14231 consensus_queue.cc:1035] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.818012 14943 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:15.818634 12036 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:15.820564 12430 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:15.821353 11887 catalog_manager.cc:5484] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
W20240503 14:00:15.821466 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e92d8f55591346158b9583feb958a8ca. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:15.838045 12036 consensus_queue.cc:237] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:15.839746 12428 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.840332 12296 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.840807 14977 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.841315 14977 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.845773 14227 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.846079 12298 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.847694 12296 consensus_queue.cc:237] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
W20240503 14:00:15.849524 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2e9f6eecbec45988b290775277dbddc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:15.849709 12166 raft_consensus.cc:1261] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.850000 12428 raft_consensus.cc:1261] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.850132 12430 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.850360 14943 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.850421 11890 catalog_manager.cc:5484] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.854282 14943 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:15.854480 11973 consensus_peers.cc:485] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5c06806a19e4f4592d26ed4acc651cd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:15.858361 14943 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:15.862715 11890 catalog_manager.cc:5484] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:15.863968 12430 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:15.865273 12166 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:15.870904 12167 consensus_queue.cc:237] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:15.873076 12296 raft_consensus.cc:1261] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.873605 12036 raft_consensus.cc:1261] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.876013 15083 consensus_queue.cc:1035] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.876405 15083 consensus_queue.cc:1035] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.880717 12037 consensus_queue.cc:237] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:15.882444 12296 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.882478 12167 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.883136 14227 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.883459 14227 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:15.883872 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4f9cb87c70e43dbb0d850c275ebb8a5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:15.885944 15081 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.886334 14896 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
W20240503 14:00:15.887248 11971 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e6ca408c4254678b67a74f49a34881b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:15.888592 12298 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.888614 12167 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:15.890214 12296 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:15.891906 11890 catalog_manager.cc:5484] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:15.893543 12036 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.895298 11893 catalog_manager.cc:5484] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:15.916922 12036 consensus_queue.cc:237] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:15.919234 12296 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.919364 12167 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.919819 15100 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.920173 15100 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:15.925365 11972 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2089eeda1e0f4ff5926584d472e6631d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:15.927062 15100 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:15.929157 12167 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:15.929387 12296 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:15.932766 11888 catalog_manager.cc:5484] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:15.937777 12036 consensus_queue.cc:237] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:15.939415 12167 raft_consensus.cc:1261] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.939492 12430 raft_consensus.cc:1261] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.939944 14977 consensus_queue.cc:1035] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:15.940327 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcdc3ae5232a49cdb003d9a2e3cf6d92. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:15.940797 14227 consensus_queue.cc:1035] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.947715 14227 raft_consensus.cc:2932] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:15.950331 12430 raft_consensus.cc:2932] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:15.950528 12167 raft_consensus.cc:2932] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:15.953051 11887 catalog_manager.cc:5484] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:15.978948 12296 consensus_queue.cc:237] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
W20240503 14:00:15.981029 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0106f7a40cb467eb4738890ac4e22d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:15.981853 12430 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.982121 12167 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.982717 15067 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.983034 15067 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.993782 12296 consensus_queue.cc:237] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:15.996317 12036 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.996820 14231 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:15.997645 12428 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:15.997994 14231 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:16.000523 12233 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92506cde7b084b14b88d4a915731eeb3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.002235 15067 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.004010 12167 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.006160 12430 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.009119 11886 catalog_manager.cc:5484] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.009455 12296 consensus_queue.cc:237] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.010812 12036 raft_consensus.cc:1261] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.011605 12430 raft_consensus.cc:1261] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.011620 14943 consensus_queue.cc:1035] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:16.013083 12233 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79b21b1e5292488aad442060b417f3c1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.013244 14943 consensus_queue.cc:1035] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.016230 14943 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.019224 12036 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.019553 12430 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.019657 11895 catalog_manager.cc:5484] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.031119 13674 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.034161 12430 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.034344 12036 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.036136 11889 catalog_manager.cc:5484] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.039297 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:16.045845 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.045871 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:16.050761 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:16.061681 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.066190 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.077344 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.079299 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.087330 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:16.096246 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:16.107522 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.110193 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.118228 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:16.118351 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.127091 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:16.133738 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:16.156385 12430 consensus_queue.cc:237] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.158828 12036 raft_consensus.cc:1261] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.159917 12167 raft_consensus.cc:1261] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.160486 13682 consensus_queue.cc:1035] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.160781 13682 consensus_queue.cc:1035] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.168900 13682 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.175091 12167 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.177306 11895 catalog_manager.cc:5484] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.180143 12036 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
W20240503 14:00:16.182576 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11a8c270532947ba9b505aace44b56f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.194490 12036 consensus_queue.cc:237] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:16.196110 12430 raft_consensus.cc:1261] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.196563 12167 raft_consensus.cc:1261] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.196604 14977 consensus_queue.cc:1035] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:16.196771 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d2e1517a8ac4041bfb4b90634cb0fa6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.197067 14977 consensus_queue.cc:1035] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.204334 14977 raft_consensus.cc:2932] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.204672 12430 raft_consensus.cc:2932] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.207417 11889 catalog_manager.cc:5484] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.208178 12167 raft_consensus.cc:2932] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.215556 12036 consensus_queue.cc:237] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:16.216888 12430 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.217092 12167 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:16.217154 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4b4e16598b740cb9d690c2442a3e994. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.217361 15100 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.217517 14896 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.222211 15100 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.222622 12430 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.224601 12167 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.224790 12036 consensus_queue.cc:237] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:16.225138 11893 catalog_manager.cc:5484] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.229465 12296 raft_consensus.cc:1261] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:16.229668 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d837845c6f644bca83315c9f7c45246d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.229923 14896 consensus_queue.cc:1035] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.231088 12167 raft_consensus.cc:1261] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.231521 14896 consensus_queue.cc:1035] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.234876 14896 raft_consensus.cc:2932] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.236263 12167 raft_consensus.cc:2932] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.238981 11893 catalog_manager.cc:5484] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.240303 12296 raft_consensus.cc:2932] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.243752 12430 consensus_queue.cc:237] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:16.246146 12036 raft_consensus.cc:1261] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:16.246610 12167 raft_consensus.cc:1261] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:00:16.247110 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11a8c270532947ba9b505aace44b56f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.247180 13682 consensus_queue.cc:1035] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:16.247480 13682 consensus_queue.cc:1035] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:16.250279 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11a8c270532947ba9b505aace44b56f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.251099 15287 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.251582 12036 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.254447 11889 catalog_manager.cc:5484] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 4 to 5, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.257903 12167 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:16.261580 12296 consensus_queue.cc:237] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.263432 12430 raft_consensus.cc:1261] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:16.263538 12233 consensus_peers.cc:485] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3481715d755a4277b0905b0832fad4af. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.263475 12036 consensus_queue.cc:237] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:16.263815 15395 ts_tablet_manager.cc:902] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:16.264066 14231 consensus_queue.cc:1035] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.264783 15395 tablet_copy_client.cc:308] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36263
I20240503 14:00:16.265470 12428 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.265897 14227 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.266049 12167 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:16.266065 11972 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c4f3bc64e7145e2b692beab730d1496. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.266686 14227 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.266783 12167 raft_consensus.cc:1261] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.267975 15233 consensus_queue.cc:1035] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.269830 14227 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.271871 12428 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.274297 11889 catalog_manager.cc:5484] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.274390 12167 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.279412 15415 ts_tablet_manager.cc:902] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:16.279925 15415 tablet_copy_client.cc:308] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:16.281653 12188 tablet_copy_service.cc:140] P 8b17d706d2344dbaae13cda9b0733143: Received BeginTabletCopySession request for tablet f4f9cb87c70e43dbb0d850c275ebb8a5 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:49327)
I20240503 14:00:16.281785 12188 tablet_copy_service.cc:161] P 8b17d706d2344dbaae13cda9b0733143: Beginning new tablet copy session on tablet f4f9cb87c70e43dbb0d850c275ebb8a5 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:49327: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-f4f9cb87c70e43dbb0d850c275ebb8a5
I20240503 14:00:16.283442 12188 tablet_copy_source_session.cc:205] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.284031 12296 consensus_queue.cc:237] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.284449 15233 raft_consensus.cc:2932] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.285972 12167 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.287307 13674 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.287267 11893 catalog_manager.cc:5484] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.287809 12036 consensus_queue.cc:237] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
W20240503 14:00:16.288059 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6a89230883347d4a64d96d321247012. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.288520 12430 raft_consensus.cc:2932] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.289393 12168 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.289570 12166 raft_consensus.cc:2932] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.290223 14977 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.290531 12428 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:16.291118 11971 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faa0a6ca00d442eaaf9fca7fc1259ec7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.291867 14943 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.292518 12428 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.293174 14977 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.295095 14943 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.297784 12167 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.299615 12428 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.300432 11895 catalog_manager.cc:5484] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.305871 15395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f9cb87c70e43dbb0d850c275ebb8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.306994 12036 consensus_queue.cc:237] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.309087 12296 raft_consensus.cc:1261] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:16.309347 11971 consensus_peers.cc:485] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a608f683c4f41f3bbdfc9b4c219ce49. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.309688 14896 consensus_queue.cc:1035] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.310053 12428 raft_consensus.cc:1261] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.310936 14896 consensus_queue.cc:1035] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.312970 12036 consensus_queue.cc:237] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.313201 15438 ts_tablet_manager.cc:902] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:16.314075 15438 tablet_copy_client.cc:308] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
W20240503 14:00:16.314859 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d2e1517a8ac4041bfb4b90634cb0fa6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.315086 12430 raft_consensus.cc:1261] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:16.315099 12166 raft_consensus.cc:1261] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:16.314877 15100 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.315307 12167 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.315479 14227 consensus_queue.cc:1035] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:16.315577 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet d1e9155230c74e92b6ee441de713a419 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:51943)
I20240503 14:00:16.315704 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet d1e9155230c74e92b6ee441de713a419 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:51943: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-d1e9155230c74e92b6ee441de713a419
I20240503 14:00:16.317083 12319 tablet_copy_source_session.cc:205] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 14:00:16.317282 11971 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d2e1517a8ac4041bfb4b90634cb0fa6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.317595 12429 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.319705 15100 consensus_queue.cc:1035] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:16.319708 11889 catalog_manager.cc:5484] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.321585 15395 tablet_copy_client.cc:783] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.321844 15395 tablet_copy_client.cc:647] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.322109 14896 raft_consensus.cc:2932] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.323848 14896 raft_consensus.cc:2932] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.327692 12296 raft_consensus.cc:2932] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.328182 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet c2e9f6eecbec45988b290775277dbddc from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:41157)
I20240503 14:00:16.328270 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet c2e9f6eecbec45988b290775277dbddc from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:41157: session id = 1b4a61e8a11440e7a9f4b300d2442f75-c2e9f6eecbec45988b290775277dbddc
I20240503 14:00:16.329262 12319 tablet_copy_source_session.cc:205] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.331490 11889 catalog_manager.cc:5484] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.332908 12429 raft_consensus.cc:2932] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.333226 12428 raft_consensus.cc:2932] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.333765 15438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e9f6eecbec45988b290775277dbddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.334693 15415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e9155230c74e92b6ee441de713a419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.335065 12167 raft_consensus.cc:2932] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.336087 15492 ts_tablet_manager.cc:902] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:16.337461 15395 tablet_copy_client.cc:523] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.337867 11890 catalog_manager.cc:5484] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:16.339306 15415 tablet_copy_client.cc:783] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.339461 15415 tablet_copy_client.cc:647] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.339661 15395 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:16.343880 15415 tablet_copy_client.cc:523] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.347733 15415 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:16.347772 15395 log.cc:824] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:16.350383 15438 tablet_copy_client.cc:783] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.351027 15438 tablet_copy_client.cc:647] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.356729 15492 tablet_copy_client.cc:308] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:16.359766 15438 tablet_copy_client.cc:523] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.361613 15438 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:16.370842 15438 log.cc:824] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:16.381631 12167 consensus_queue.cc:237] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:16.383610 12036 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.383663 12296 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.384016 15081 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.384227 13658 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.390173 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet 2089eeda1e0f4ff5926584d472e6631d from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:60777)
I20240503 14:00:16.390319 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet 2089eeda1e0f4ff5926584d472e6631d from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:60777: session id = b67f337e10994234baedbf0b19c13513-2089eeda1e0f4ff5926584d472e6631d
I20240503 14:00:16.391834 12057 tablet_copy_source_session.cc:205] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.395921 15492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2089eeda1e0f4ff5926584d472e6631d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.399942 12428 consensus_queue.cc:237] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.400511 15540 ts_tablet_manager.cc:902] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:16.402146 12298 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.402140 12167 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.402698 15288 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.403060 15288 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:16.403556 15540 tablet_copy_client.cc:308] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
W20240503 14:00:16.403959 12365 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8933689df2c45b59a9cd16024ddb7d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.404067 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet e92d8f55591346158b9583feb958a8ca from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:41157)
I20240503 14:00:16.404168 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet e92d8f55591346158b9583feb958a8ca from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:41157: session id = 1b4a61e8a11440e7a9f4b300d2442f75-e92d8f55591346158b9583feb958a8ca
I20240503 14:00:16.405117 12319 tablet_copy_source_session.cc:205] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.406762 15492 tablet_copy_client.cc:783] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.406989 15492 tablet_copy_client.cc:647] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
W20240503 14:00:16.407223 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c42fee9fadbb4601a7fe32eb6a04e767. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.409338 15492 tablet_copy_client.cc:523] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.411581 15492 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:16.413502 15438 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.413898 15438 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:16.414252 15438 ts_tablet_manager.cc:1367] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.053s user 0.035s sys 0.004s
I20240503 14:00:16.416715 15438 raft_consensus.cc:348] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.416955 15438 raft_consensus.cc:729] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.417435 15438 consensus_queue.cc:260] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.418368 15438 ts_tablet_manager.cc:1398] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:00:16.419147 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-c2e9f6eecbec45988b290775277dbddc received from {username='slave'} at 127.0.99.136:41157
I20240503 14:00:16.419248 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-c2e9f6eecbec45988b290775277dbddc on tablet c2e9f6eecbec45988b290775277dbddc with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:16.420419 15492 log.cc:824] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:16.420570 15395 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.420924 15395 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:16.421244 15395 ts_tablet_manager.cc:1367] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.082s user 0.031s sys 0.004s
I20240503 14:00:16.421952 15540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92d8f55591346158b9583feb958a8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.423753 15395 raft_consensus.cc:348] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:16.423985 15395 raft_consensus.cc:729] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.424463 15395 consensus_queue.cc:260] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:16.425159 15540 tablet_copy_client.cc:783] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.425341 15540 tablet_copy_client.cc:647] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.425391 15395 ts_tablet_manager.cc:1398] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 14:00:16.425956 15415 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.426277 15415 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:16.426491 15415 ts_tablet_manager.cc:1367] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.079s user 0.027s sys 0.000s
I20240503 14:00:16.427390 12188 tablet_copy_service.cc:342] P 8b17d706d2344dbaae13cda9b0733143: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-f4f9cb87c70e43dbb0d850c275ebb8a5 received from {username='slave'} at 127.0.99.135:49327
I20240503 14:00:16.427492 12188 tablet_copy_service.cc:433] P 8b17d706d2344dbaae13cda9b0733143: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-f4f9cb87c70e43dbb0d850c275ebb8a5 on tablet f4f9cb87c70e43dbb0d850c275ebb8a5 with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:16.428431 15415 raft_consensus.cc:348] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:16.428570 15415 raft_consensus.cc:729] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.428741 15415 consensus_queue.cc:260] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:16.428967 15415 ts_tablet_manager.cc:1398] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:16.429735 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-d1e9155230c74e92b6ee441de713a419 received from {username='slave'} at 127.0.99.135:51943
I20240503 14:00:16.429816 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-d1e9155230c74e92b6ee441de713a419 on tablet d1e9155230c74e92b6ee441de713a419 with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:16.431079 15551 ts_tablet_manager.cc:902] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:16.431946 15551 tablet_copy_client.cc:308] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:16.432655 15415 ts_tablet_manager.cc:902] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:16.433036 15415 tablet_copy_client.cc:308] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:16.434093 15540 tablet_copy_client.cc:523] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.435561 15540 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:16.438946 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet f5c06806a19e4f4592d26ed4acc651cd from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:46961)
I20240503 14:00:16.439054 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet f5c06806a19e4f4592d26ed4acc651cd from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:46961: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-f5c06806a19e4f4592d26ed4acc651cd
I20240503 14:00:16.440068 12057 tablet_copy_source_session.cc:205] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.440822 12036 consensus_queue.cc:237] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:16.440869 15081 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.443132 12167 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:16.443624 12428 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:00:16.443827 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4b4e16598b740cb9d690c2442a3e994. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:16.444046 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4b4e16598b740cb9d690c2442a3e994. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.444312 14227 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:16.444411 12036 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.445014 11895 catalog_manager.cc:5484] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.445976 12296 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.448505 15100 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:16.449959 14227 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.451026 12167 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.451964 12428 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.455112 11895 catalog_manager.cc:5484] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:16.456656 15382 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.457444 15580 ts_tablet_manager.cc:902] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:16.457975 15580 tablet_copy_client.cc:308] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:16.458200 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet 92506cde7b084b14b88d4a915731eeb3 from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:38411)
I20240503 14:00:16.458310 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet 92506cde7b084b14b88d4a915731eeb3 from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:38411: session id = 66190085d1c04e068d5c0dec9c4bf07d-92506cde7b084b14b88d4a915731eeb3
I20240503 14:00:16.459214 12319 tablet_copy_source_session.cc:205] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.461112 12167 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.461334 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet dcdc3ae5232a49cdb003d9a2e3cf6d92 from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:60777)
I20240503 14:00:16.461432 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet dcdc3ae5232a49cdb003d9a2e3cf6d92 from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:60777: session id = b67f337e10994234baedbf0b19c13513-dcdc3ae5232a49cdb003d9a2e3cf6d92
I20240503 14:00:16.462283 12057 tablet_copy_source_session.cc:205] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.463613 15415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c06806a19e4f4592d26ed4acc651cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.463474 12296 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.465116 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcdc3ae5232a49cdb003d9a2e3cf6d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.467078 11889 catalog_manager.cc:5484] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.468565 15580 tablet_copy_client.cc:783] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.468577 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92506cde7b084b14b88d4a915731eeb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.468791 15580 tablet_copy_client.cc:647] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.470397 15580 tablet_copy_client.cc:523] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.471614 15415 tablet_copy_client.cc:783] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.471791 15415 tablet_copy_client.cc:647] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.472266 15580 tablet_bootstrap.cc:492] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:16.480751 15415 tablet_copy_client.cc:523] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.482074 15415 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:16.485864 15438 ts_tablet_manager.cc:902] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:16.486331 15438 tablet_copy_client.cc:308] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:16.486878 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet a0106f7a40cb467eb4738890ac4e22d4 from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:41157)
I20240503 14:00:16.486994 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet a0106f7a40cb467eb4738890ac4e22d4 from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:41157: session id = 1b4a61e8a11440e7a9f4b300d2442f75-a0106f7a40cb467eb4738890ac4e22d4
I20240503 14:00:16.487948 12319 tablet_copy_source_session.cc:205] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.488799 15438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0106f7a40cb467eb4738890ac4e22d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.489076 15551 tablet_copy_client.cc:783] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.489295 15551 tablet_copy_client.cc:647] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.490479 15598 ts_tablet_manager.cc:902] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:16.490851 15551 tablet_copy_client.cc:523] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.491044 15598 tablet_copy_client.cc:308] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:16.491350 15438 tablet_copy_client.cc:783] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.491556 15438 tablet_copy_client.cc:647] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.498581 15438 tablet_copy_client.cc:523] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.499958 15438 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:16.507086 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet 5e6ca408c4254678b67a74f49a34881b from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:53301)
I20240503 14:00:16.507207 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet 5e6ca408c4254678b67a74f49a34881b from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:53301: session id = 66190085d1c04e068d5c0dec9c4bf07d-5e6ca408c4254678b67a74f49a34881b
I20240503 14:00:16.508138 12057 tablet_copy_source_session.cc:205] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.510960 15598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6ca408c4254678b67a74f49a34881b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.512187 15551 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:16.513615 15598 tablet_copy_client.cc:783] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.513830 15598 tablet_copy_client.cc:647] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.521131 15598 tablet_copy_client.cc:523] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.522675 15598 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:16.524730 15492 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.524936 15540 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.525239 15540 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:16.525246 15492 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:16.525408 15540 ts_tablet_manager.cc:1367] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.090s user 0.027s sys 0.001s
I20240503 14:00:16.525712 15492 ts_tablet_manager.cc:1367] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.114s user 0.034s sys 0.005s
I20240503 14:00:16.525938 15540 raft_consensus.cc:348] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.526051 15540 raft_consensus.cc:729] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.526201 15540 consensus_queue.cc:260] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.526612 15540 ts_tablet_manager.cc:1398] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:16.527292 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-e92d8f55591346158b9583feb958a8ca received from {username='slave'} at 127.0.99.136:41157
I20240503 14:00:16.527381 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-e92d8f55591346158b9583feb958a8ca on tablet e92d8f55591346158b9583feb958a8ca with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:16.533301 15551 log.cc:824] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:00:16.540551 15415 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.540824 15415 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:16.541018 15415 ts_tablet_manager.cc:1367] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.059s user 0.026s sys 0.000s
I20240503 14:00:16.541601 15415 raft_consensus.cc:348] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:16.541733 15415 raft_consensus.cc:729] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.541899 15415 consensus_queue.cc:260] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:16.541872 15492 raft_consensus.cc:348] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:16.542287 15492 raft_consensus.cc:729] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.543010 15492 consensus_queue.cc:260] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:16.545040 15415 ts_tablet_manager.cc:1398] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:00:16.546509 15492 ts_tablet_manager.cc:1398] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20240503 14:00:16.546862 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-f5c06806a19e4f4592d26ed4acc651cd received from {username='slave'} at 127.0.99.135:46961
I20240503 14:00:16.546959 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-f5c06806a19e4f4592d26ed4acc651cd on tablet f5c06806a19e4f4592d26ed4acc651cd with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:16.548535 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-2089eeda1e0f4ff5926584d472e6631d received from {username='slave'} at 127.0.99.134:60777
I20240503 14:00:16.548612 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session b67f337e10994234baedbf0b19c13513-2089eeda1e0f4ff5926584d472e6631d on tablet 2089eeda1e0f4ff5926584d472e6631d with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:16.556697 15580 tablet_bootstrap.cc:492] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.557056 15580 tablet_bootstrap.cc:492] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:16.557226 15580 ts_tablet_manager.cc:1367] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.085s user 0.028s sys 0.000s
I20240503 14:00:16.557749 15580 raft_consensus.cc:348] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:16.557868 15580 raft_consensus.cc:729] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.558035 15580 consensus_queue.cc:260] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:16.558794 15580 ts_tablet_manager.cc:1398] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:16.559435 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-dcdc3ae5232a49cdb003d9a2e3cf6d92 received from {username='slave'} at 127.0.99.134:60777
I20240503 14:00:16.559513 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session b67f337e10994234baedbf0b19c13513-dcdc3ae5232a49cdb003d9a2e3cf6d92 on tablet dcdc3ae5232a49cdb003d9a2e3cf6d92 with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:16.595889 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.611322 15438 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.611665 15438 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:16.611845 15438 ts_tablet_manager.cc:1367] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.112s user 0.026s sys 0.001s
I20240503 14:00:16.612085 12036 consensus_queue.cc:237] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.612308 15438 raft_consensus.cc:348] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.612423 15438 raft_consensus.cc:729] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.613684 15438 consensus_queue.cc:260] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.614023 15438 ts_tablet_manager.cc:1398] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:16.614746 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-a0106f7a40cb467eb4738890ac4e22d4 received from {username='slave'} at 127.0.99.136:41157
I20240503 14:00:16.614832 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-a0106f7a40cb467eb4738890ac4e22d4 on tablet a0106f7a40cb467eb4738890ac4e22d4 with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:16.614811 14037 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:16.615868 12428 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:16.616137 15551 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.616272 12296 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:16.616564 15551 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:16.616905 15551 ts_tablet_manager.cc:1367] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.118s user 0.031s sys 0.000s
I20240503 14:00:16.617480 14896 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:16.618229 14977 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:16.618510 14977 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:16.620568 15551 raft_consensus.cc:348] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.620805 15551 raft_consensus.cc:729] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.621351 15551 consensus_queue.cc:260] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.622205 15551 ts_tablet_manager.cc:1398] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:00:16.622814 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-92506cde7b084b14b88d4a915731eeb3 received from {username='slave'} at 127.0.99.133:38411
I20240503 14:00:16.622879 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-92506cde7b084b14b88d4a915731eeb3 on tablet 92506cde7b084b14b88d4a915731eeb3 with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:16.623821 15551 ts_tablet_manager.cc:902] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:16.624168 15551 tablet_copy_client.cc:308] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:16.624491 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet 79b21b1e5292488aad442060b417f3c1 from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:38411)
I20240503 14:00:16.624567 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet 79b21b1e5292488aad442060b417f3c1 from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:38411: session id = 66190085d1c04e068d5c0dec9c4bf07d-79b21b1e5292488aad442060b417f3c1
I20240503 14:00:16.625394 12319 tablet_copy_source_session.cc:205] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.626420 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b21b1e5292488aad442060b417f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.626279 14896 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.629252 15551 tablet_copy_client.cc:783] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.629173 12428 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.629397 15551 tablet_copy_client.cc:647] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.629341 11889 catalog_manager.cc:5484] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 4 to 5, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.631163 12296 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.636353 15438 ts_tablet_manager.cc:902] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:16.636826 15551 tablet_copy_client.cc:523] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.638106 15551 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:16.644989 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:16.645316 15438 tablet_copy_client.cc:308] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33927
I20240503 14:00:16.652045 15415 ts_tablet_manager.cc:902] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:16.652494 15415 tablet_copy_client.cc:308] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:16.654242 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet d837845c6f644bca83315c9f7c45246d from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:46961)
I20240503 14:00:16.654372 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet d837845c6f644bca83315c9f7c45246d from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:46961: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-d837845c6f644bca83315c9f7c45246d
I20240503 14:00:16.655355 12057 tablet_copy_source_session.cc:205] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.659883 15415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d837845c6f644bca83315c9f7c45246d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.662320 15415 tablet_copy_client.cc:783] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.662488 15415 tablet_copy_client.cc:647] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.663326 15598 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.663700 15598 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:16.663939 15598 ts_tablet_manager.cc:1367] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.141s user 0.035s sys 0.000s
I20240503 14:00:16.664130 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.667059 15551 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.667359 15551 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:16.667521 15551 ts_tablet_manager.cc:1367] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.002s
I20240503 14:00:16.673658 12450 tablet_copy_service.cc:140] P 84c63c6f551949f6a1d7c07760824361: Received BeginTabletCopySession request for tablet 11a8c270532947ba9b505aace44b56f6 from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:55937)
I20240503 14:00:16.673831 12450 tablet_copy_service.cc:161] P 84c63c6f551949f6a1d7c07760824361: Beginning new tablet copy session on tablet 11a8c270532947ba9b505aace44b56f6 from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:55937: session id = 1b4a61e8a11440e7a9f4b300d2442f75-11a8c270532947ba9b505aace44b56f6
I20240503 14:00:16.675436 12450 tablet_copy_source_session.cc:205] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.676817 15438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11a8c270532947ba9b505aace44b56f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.676949 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:16.679688 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.681143 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:16.683672 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:16.689425 15438 tablet_copy_client.cc:783] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.689617 15438 tablet_copy_client.cc:647] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.690130 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.697806 15438 tablet_copy_client.cc:523] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.704166 15438 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:16.707476 15580 ts_tablet_manager.cc:902] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:16.708781 15580 tablet_copy_client.cc:308] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:16.709220 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet 6c4f3bc64e7145e2b692beab730d1496 from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:60777)
I20240503 14:00:16.709318 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet 6c4f3bc64e7145e2b692beab730d1496 from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:60777: session id = b67f337e10994234baedbf0b19c13513-6c4f3bc64e7145e2b692beab730d1496
I20240503 14:00:16.710156 12057 tablet_copy_source_session.cc:205] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.711084 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4f3bc64e7145e2b692beab730d1496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.711444 15415 tablet_copy_client.cc:523] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.713521 15580 tablet_copy_client.cc:783] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.713671 15580 tablet_copy_client.cc:647] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.717201 15415 tablet_bootstrap.cc:492] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:16.730446 15492 ts_tablet_manager.cc:902] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:16.730953 15492 tablet_copy_client.cc:308] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:16.731420 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet 7d2e1517a8ac4041bfb4b90634cb0fa6 from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:60777)
I20240503 14:00:16.731519 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet 7d2e1517a8ac4041bfb4b90634cb0fa6 from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:60777: session id = b67f337e10994234baedbf0b19c13513-7d2e1517a8ac4041bfb4b90634cb0fa6
I20240503 14:00:16.732383 12057 tablet_copy_source_session.cc:205] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.733239 15492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2e1517a8ac4041bfb4b90634cb0fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.735905 15492 tablet_copy_client.cc:783] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.736112 15492 tablet_copy_client.cc:647] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.736369 15580 tablet_copy_client.cc:523] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.737576 15580 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:16.739589 14037 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:16.740114 14168 raft_consensus.cc:1203] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:16.745157 15540 ts_tablet_manager.cc:902] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:16.745632 15540 tablet_copy_client.cc:308] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:16.746064 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet a6a89230883347d4a64d96d321247012 from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:41157)
I20240503 14:00:16.746176 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet a6a89230883347d4a64d96d321247012 from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:41157: session id = 1b4a61e8a11440e7a9f4b300d2442f75-a6a89230883347d4a64d96d321247012
I20240503 14:00:16.746626 15551 raft_consensus.cc:348] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.746778 15551 raft_consensus.cc:729] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.746907 15551 consensus_queue.cc:260] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.747308 15551 ts_tablet_manager.cc:1398] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.080s user 0.000s sys 0.001s
I20240503 14:00:16.747501 12319 tablet_copy_source_session.cc:205] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.746626 15598 raft_consensus.cc:348] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.747682 15598 raft_consensus.cc:729] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.747804 15598 consensus_queue.cc:260] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.748196 15598 ts_tablet_manager.cc:1398] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.084s user 0.001s sys 0.000s
I20240503 14:00:16.748553 15540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a89230883347d4a64d96d321247012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.748620 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-5e6ca408c4254678b67a74f49a34881b received from {username='slave'} at 127.0.99.133:53301
I20240503 14:00:16.748708 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-5e6ca408c4254678b67a74f49a34881b on tablet 5e6ca408c4254678b67a74f49a34881b with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:16.750799 15540 tablet_copy_client.cc:783] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.750962 15540 tablet_copy_client.cc:647] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.751098 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-79b21b1e5292488aad442060b417f3c1 received from {username='slave'} at 127.0.99.133:38411
I20240503 14:00:16.751163 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-79b21b1e5292488aad442060b417f3c1 on tablet 79b21b1e5292488aad442060b417f3c1 with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:16.751943 15551 ts_tablet_manager.cc:902] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:16.752334 15551 tablet_copy_client.cc:308] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:16.752775 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet faa0a6ca00d442eaaf9fca7fc1259ec7 from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:53301)
I20240503 14:00:16.752882 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet faa0a6ca00d442eaaf9fca7fc1259ec7 from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:53301: session id = 66190085d1c04e068d5c0dec9c4bf07d-faa0a6ca00d442eaaf9fca7fc1259ec7
I20240503 14:00:16.753726 12057 tablet_copy_source_session.cc:205] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.754920 15540 tablet_copy_client.cc:523] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.756229 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa0a6ca00d442eaaf9fca7fc1259ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.756464 15540 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:16.758348 15551 tablet_copy_client.cc:783] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.758496 15551 tablet_copy_client.cc:647] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.761178 15492 tablet_copy_client.cc:523] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.762102 15438 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.762342 15438 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:16.762467 15492 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:16.762499 15438 ts_tablet_manager.cc:1367] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.058s user 0.020s sys 0.004s
I20240503 14:00:16.762912 15438 raft_consensus.cc:348] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:16.763020 15438 raft_consensus.cc:729] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.763147 15438 consensus_queue.cc:260] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:16.763520 15438 ts_tablet_manager.cc:1398] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 14:00:16.765517 12450 tablet_copy_service.cc:342] P 84c63c6f551949f6a1d7c07760824361: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-11a8c270532947ba9b505aace44b56f6 received from {username='slave'} at 127.0.99.136:55937
I20240503 14:00:16.765606 12450 tablet_copy_service.cc:433] P 84c63c6f551949f6a1d7c07760824361: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-11a8c270532947ba9b505aace44b56f6 on tablet 11a8c270532947ba9b505aace44b56f6 with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:16.766405 15551 tablet_copy_client.cc:523] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.767715 15551 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:16.774041 15580 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.774310 15580 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:16.774528 15580 ts_tablet_manager.cc:1367] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.037s user 0.024s sys 0.005s
I20240503 14:00:16.774963 15580 raft_consensus.cc:348] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:16.775067 15580 raft_consensus.cc:729] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.775223 15580 consensus_queue.cc:260] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
W20240503 14:00:16.776829 11972 consensus_peers.cc:485] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5c06806a19e4f4592d26ed4acc651cd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.778213 15580 ts_tablet_manager.cc:1398] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 14:00:16.780591 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-6c4f3bc64e7145e2b692beab730d1496 received from {username='slave'} at 127.0.99.134:60777
I20240503 14:00:16.780669 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session b67f337e10994234baedbf0b19c13513-6c4f3bc64e7145e2b692beab730d1496 on tablet 6c4f3bc64e7145e2b692beab730d1496 with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:16.795910 15492 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.796214 15492 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:16.796393 15492 ts_tablet_manager.cc:1367] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.007s
I20240503 14:00:16.796831 15492 raft_consensus.cc:348] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.796957 15492 raft_consensus.cc:729] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.797102 15492 consensus_queue.cc:260] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.797827 15492 ts_tablet_manager.cc:1398] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:16.798230 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-7d2e1517a8ac4041bfb4b90634cb0fa6 received from {username='slave'} at 127.0.99.134:60777
I20240503 14:00:16.798316 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session b67f337e10994234baedbf0b19c13513-7d2e1517a8ac4041bfb4b90634cb0fa6 on tablet 7d2e1517a8ac4041bfb4b90634cb0fa6 with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:16.799127 14037 raft_consensus.cc:1203] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:16.807126 15551 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.807480 15551 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:16.807686 15551 ts_tablet_manager.cc:1367] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.040s user 0.023s sys 0.003s
I20240503 14:00:16.808264 15551 raft_consensus.cc:348] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.808387 15551 raft_consensus.cc:729] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.808538 15551 consensus_queue.cc:260] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.809458 15551 ts_tablet_manager.cc:1398] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:16.809885 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-faa0a6ca00d442eaaf9fca7fc1259ec7 received from {username='slave'} at 127.0.99.133:53301
I20240503 14:00:16.809968 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-faa0a6ca00d442eaaf9fca7fc1259ec7 on tablet faa0a6ca00d442eaaf9fca7fc1259ec7 with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:16.811962 15551 ts_tablet_manager.cc:902] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:16.819303 15688 ts_tablet_manager.cc:902] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:16.819871 15688 tablet_copy_client.cc:308] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:16.821349 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet 4a608f683c4f41f3bbdfc9b4c219ce49 from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:53301)
I20240503 14:00:16.821457 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet 4a608f683c4f41f3bbdfc9b4c219ce49 from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:53301: session id = 66190085d1c04e068d5c0dec9c4bf07d-4a608f683c4f41f3bbdfc9b4c219ce49
I20240503 14:00:16.822353 12057 tablet_copy_source_session.cc:205] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.823611 15438 ts_tablet_manager.cc:902] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:16.824012 15438 tablet_copy_client.cc:308] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33927
I20240503 14:00:16.824546 12450 tablet_copy_service.cc:140] P 84c63c6f551949f6a1d7c07760824361: Received BeginTabletCopySession request for tablet c8933689df2c45b59a9cd16024ddb7d9 from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:55937)
I20240503 14:00:16.824653 12450 tablet_copy_service.cc:161] P 84c63c6f551949f6a1d7c07760824361: Beginning new tablet copy session on tablet c8933689df2c45b59a9cd16024ddb7d9 from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:55937: session id = 1b4a61e8a11440e7a9f4b300d2442f75-c8933689df2c45b59a9cd16024ddb7d9
I20240503 14:00:16.824987 15688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a608f683c4f41f3bbdfc9b4c219ce49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.825611 12450 tablet_copy_source_session.cc:205] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.826403 15438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8933689df2c45b59a9cd16024ddb7d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.828678 14037 raft_consensus.cc:1203] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:16.829538 15438 tablet_copy_client.cc:783] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.829746 15438 tablet_copy_client.cc:647] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.829926 15598 ts_tablet_manager.cc:902] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:16.830521 15598 tablet_copy_client.cc:308] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:16.831086 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet 3481715d755a4277b0905b0832fad4af from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:38411)
I20240503 14:00:16.831197 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet 3481715d755a4277b0905b0832fad4af from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:38411: session id = 66190085d1c04e068d5c0dec9c4bf07d-3481715d755a4277b0905b0832fad4af
I20240503 14:00:16.832159 12319 tablet_copy_source_session.cc:205] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.832654 15688 tablet_copy_client.cc:783] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.832859 15688 tablet_copy_client.cc:647] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.833189 15598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3481715d755a4277b0905b0832fad4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.836360 15598 tablet_copy_client.cc:783] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.836565 15598 tablet_copy_client.cc:647] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.838685 15598 tablet_copy_client.cc:523] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.840456 15598 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:16.812410 15551 tablet_copy_client.cc:308] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:16.842384 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet 7d2e1517a8ac4041bfb4b90634cb0fa6 from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:53301)
I20240503 14:00:16.842514 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet 7d2e1517a8ac4041bfb4b90634cb0fa6 from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:53301: session id = 66190085d1c04e068d5c0dec9c4bf07d-7d2e1517a8ac4041bfb4b90634cb0fa6
I20240503 14:00:16.843492 12057 tablet_copy_source_session.cc:205] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.845578 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2e1517a8ac4041bfb4b90634cb0fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.847697 15551 tablet_copy_client.cc:783] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.847834 15551 tablet_copy_client.cc:647] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.848932 15395 ts_tablet_manager.cc:902] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:16.849494 15395 tablet_copy_client.cc:308] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33927
I20240503 14:00:16.851876 15438 tablet_copy_client.cc:523] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.852208 13906 raft_consensus.cc:1203] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:16.853493 15551 tablet_copy_client.cc:523] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.854816 15551 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:16.859063 15540 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.859442 15540 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:16.859619 15540 ts_tablet_manager.cc:1367] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.103s user 0.029s sys 0.000s
I20240503 14:00:16.860106 15540 raft_consensus.cc:348] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.860229 15540 raft_consensus.cc:729] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.860457 15540 consensus_queue.cc:260] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.861037 15438 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:16.861074 12450 tablet_copy_service.cc:140] P 84c63c6f551949f6a1d7c07760824361: Received BeginTabletCopySession request for tablet 11a8c270532947ba9b505aace44b56f6 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:43887)
I20240503 14:00:16.861161 12450 tablet_copy_service.cc:161] P 84c63c6f551949f6a1d7c07760824361: Beginning new tablet copy session on tablet 11a8c270532947ba9b505aace44b56f6 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:43887: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-11a8c270532947ba9b505aace44b56f6
I20240503 14:00:16.861281 15540 ts_tablet_manager.cc:1398] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:16.861719 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-a6a89230883347d4a64d96d321247012 received from {username='slave'} at 127.0.99.136:41157
I20240503 14:00:16.861789 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-a6a89230883347d4a64d96d321247012 on tablet a6a89230883347d4a64d96d321247012 with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:16.862138 12450 tablet_copy_source_session.cc:205] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.863101 15395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11a8c270532947ba9b505aace44b56f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.865520 15395 tablet_copy_client.cc:783] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.865700 15395 tablet_copy_client.cc:647] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.866429 15688 tablet_copy_client.cc:523] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.867858 15688 tablet_bootstrap.cc:492] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:16.879554 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.882000 15395 tablet_copy_client.cc:523] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.883498 15395 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:16.886148 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.906469 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:16.909840 13757 raft_consensus.cc:1203] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:16.912410 15492 ts_tablet_manager.cc:902] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:16.912843 15492 tablet_copy_client.cc:308] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36263
I20240503 14:00:16.916407 13757 raft_consensus.cc:1203] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:16.916602 15551 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.916899 15551 tablet_bootstrap.cc:492] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:16.917109 15551 ts_tablet_manager.cc:1367] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.062s user 0.022s sys 0.007s
I20240503 14:00:16.917546 15551 raft_consensus.cc:348] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.917675 15551 raft_consensus.cc:729] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.927702 15551 consensus_queue.cc:260] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.928109 15551 ts_tablet_manager.cc:1398] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:00:16.929337 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-7d2e1517a8ac4041bfb4b90634cb0fa6 received from {username='slave'} at 127.0.99.133:53301
I20240503 14:00:16.929423 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-7d2e1517a8ac4041bfb4b90634cb0fa6 on tablet 7d2e1517a8ac4041bfb4b90634cb0fa6 with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:16.931726 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.932516 15688 tablet_bootstrap.cc:492] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.932822 15688 tablet_bootstrap.cc:492] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:16.933020 15688 ts_tablet_manager.cc:1367] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.065s user 0.029s sys 0.000s
I20240503 14:00:16.933526 15688 raft_consensus.cc:348] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.933676 15688 raft_consensus.cc:729] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.933858 15688 consensus_queue.cc:260] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:16.934787 15688 ts_tablet_manager.cc:1398] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:16.935153 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-4a608f683c4f41f3bbdfc9b4c219ce49 received from {username='slave'} at 127.0.99.133:53301
I20240503 14:00:16.935277 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-4a608f683c4f41f3bbdfc9b4c219ce49 on tablet 4a608f683c4f41f3bbdfc9b4c219ce49 with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:16.938488 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:16.938839 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:16.939617 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:16.946839 12188 tablet_copy_service.cc:140] P 8b17d706d2344dbaae13cda9b0733143: Received BeginTabletCopySession request for tablet c42fee9fadbb4601a7fe32eb6a04e767 from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:35821)
I20240503 14:00:16.946960 12188 tablet_copy_service.cc:161] P 8b17d706d2344dbaae13cda9b0733143: Beginning new tablet copy session on tablet c42fee9fadbb4601a7fe32eb6a04e767 from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:35821: session id = b67f337e10994234baedbf0b19c13513-c42fee9fadbb4601a7fe32eb6a04e767
I20240503 14:00:16.947829 12188 tablet_copy_source_session.cc:205] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:16.948755 15492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42fee9fadbb4601a7fe32eb6a04e767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:16.950888 15492 tablet_copy_client.cc:783] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:16.951040 15492 tablet_copy_client.cc:647] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:16.952069 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.952394 15492 tablet_copy_client.cc:523] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:16.953542 15492 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:16.956017 13906 raft_consensus.cc:1203] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:16.956709 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:16.962356 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:16.980212 15492 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:16.980576 15492 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:16.980751 15492 ts_tablet_manager.cc:1367] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.000s
I20240503 14:00:16.981190 15492 raft_consensus.cc:348] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:16.981334 15492 raft_consensus.cc:729] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:16.981501 15492 consensus_queue.cc:260] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:16.981781 15492 ts_tablet_manager.cc:1398] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:16.982364 12188 tablet_copy_service.cc:342] P 8b17d706d2344dbaae13cda9b0733143: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-c42fee9fadbb4601a7fe32eb6a04e767 received from {username='slave'} at 127.0.99.134:35821
I20240503 14:00:16.982443 12188 tablet_copy_service.cc:433] P 8b17d706d2344dbaae13cda9b0733143: ending tablet copy session b67f337e10994234baedbf0b19c13513-c42fee9fadbb4601a7fe32eb6a04e767 on tablet c42fee9fadbb4601a7fe32eb6a04e767 with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:16.985472 12428 consensus_queue.cc:237] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:16.986893 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:16.987098 12167 raft_consensus.cc:1261] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.987131 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:16.987689 13682 consensus_queue.cc:1035] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:16.988335 12365 consensus_peers.cc:485] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 211de6f35a844e278e8c1eb1621e7c5f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:16.992259 12296 raft_consensus.cc:1261] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:16.993521 13682 consensus_queue.cc:1035] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.006754 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:17.015867 14168 raft_consensus.cc:1203] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.016698 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.017616 15492 ts_tablet_manager.cc:902] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:17.018105 15492 tablet_copy_client.cc:308] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:17.018548 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet a4b4e16598b740cb9d690c2442a3e994 from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:60777)
I20240503 14:00:17.018651 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet a4b4e16598b740cb9d690c2442a3e994 from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:60777: session id = b67f337e10994234baedbf0b19c13513-a4b4e16598b740cb9d690c2442a3e994
I20240503 14:00:17.019579 12057 tablet_copy_source_session.cc:205] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:17.020188 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.020882 15492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b4e16598b740cb9d690c2442a3e994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:17.022965 15492 tablet_copy_client.cc:783] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:17.023104 15492 tablet_copy_client.cc:647] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:17.023941 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.033694 15492 tablet_copy_client.cc:523] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:17.035040 15492 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:17.050356 15415 tablet_bootstrap.cc:492] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:17.050691 15415 tablet_bootstrap.cc:492] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:17.050890 15415 ts_tablet_manager.cc:1367] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.334s user 0.027s sys 0.000s
I20240503 14:00:17.051384 15415 raft_consensus.cc:348] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:17.051540 15415 raft_consensus.cc:729] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:17.053488 12036 consensus_queue.cc:237] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:17.055097 12298 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:17.055223 12428 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:17.055644 14977 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:17.055799 11971 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 355b5456887b4c5694badc9d53b1809a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.056238 14977 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.059129 15415 consensus_queue.cc:260] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:17.059381 15868 raft_consensus.cc:2932] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:17.064656 15415 ts_tablet_manager.cc:1398] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20240503 14:00:17.064958 12167 raft_consensus.cc:2932] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:17.067128 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-d837845c6f644bca83315c9f7c45246d received from {username='slave'} at 127.0.99.135:46961
I20240503 14:00:17.067256 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-d837845c6f644bca83315c9f7c45246d on tablet d837845c6f644bca83315c9f7c45246d with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:17.068228 15540 ts_tablet_manager.cc:902] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:17.068679 15540 tablet_copy_client.cc:308] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:17.068928 12296 raft_consensus.cc:2932] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:17.072587 11892 catalog_manager.cc:5484] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:17.073377 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet f5c06806a19e4f4592d26ed4acc651cd from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:40263)
I20240503 14:00:17.073472 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet f5c06806a19e4f4592d26ed4acc651cd from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:40263: session id = 1b4a61e8a11440e7a9f4b300d2442f75-f5c06806a19e4f4592d26ed4acc651cd
I20240503 14:00:17.074244 12057 tablet_copy_source_session.cc:205] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:17.075001 15540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c06806a19e4f4592d26ed4acc651cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:17.076284 14168 raft_consensus.cc:1203] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.077157 15540 tablet_copy_client.cc:783] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:17.077333 15540 tablet_copy_client.cc:647] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:17.084060 15415 ts_tablet_manager.cc:902] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:17.084519 15415 tablet_copy_client.cc:308] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:17.084928 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet a4b4e16598b740cb9d690c2442a3e994 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:46961)
I20240503 14:00:17.085024 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet a4b4e16598b740cb9d690c2442a3e994 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:46961: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-a4b4e16598b740cb9d690c2442a3e994
I20240503 14:00:17.085853 15492 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:17.085958 12057 tablet_copy_source_session.cc:205] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:17.086115 15492 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:17.086289 15492 ts_tablet_manager.cc:1367] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.051s user 0.026s sys 0.000s
I20240503 14:00:17.086799 15415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b4e16598b740cb9d690c2442a3e994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:17.087566 15492 raft_consensus.cc:348] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:17.087769 15492 raft_consensus.cc:729] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:17.087999 15492 consensus_queue.cc:260] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:17.090173 15492 ts_tablet_manager.cc:1398] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:00:17.090386 15415 tablet_copy_client.cc:783] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:17.090548 15415 tablet_copy_client.cc:647] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:17.091717 15415 tablet_copy_client.cc:523] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:17.092927 15415 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:17.095083 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-a4b4e16598b740cb9d690c2442a3e994 received from {username='slave'} at 127.0.99.134:60777
I20240503 14:00:17.095152 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session b67f337e10994234baedbf0b19c13513-a4b4e16598b740cb9d690c2442a3e994 on tablet a4b4e16598b740cb9d690c2442a3e994 with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:17.104043 14977 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:17.108048 11892 catalog_manager.cc:5484] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:17.109091 12428 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:17.112608 12298 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:17.114717 15540 tablet_copy_client.cc:523] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:17.116163 15540 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:17.134091 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.134850 13757 raft_consensus.cc:1203] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.147023 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.153743 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.154361 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.155220 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:17.156903 15395 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:17.157186 15395 tablet_bootstrap.cc:492] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:17.157359 15395 ts_tablet_manager.cc:1367] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.274s user 0.027s sys 0.000s
I20240503 14:00:17.157471 13906 raft_consensus.cc:1203] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:17.157804 15395 raft_consensus.cc:348] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:17.157913 15395 raft_consensus.cc:729] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:17.163342 15395 consensus_queue.cc:260] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:17.168445 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:17.171205 15395 ts_tablet_manager.cc:1398] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 14:00:17.173818 12450 tablet_copy_service.cc:342] P 84c63c6f551949f6a1d7c07760824361: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-11a8c270532947ba9b505aace44b56f6 received from {username='slave'} at 127.0.99.135:43887
I20240503 14:00:17.173897 12450 tablet_copy_service.cc:433] P 84c63c6f551949f6a1d7c07760824361: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-11a8c270532947ba9b505aace44b56f6 on tablet 11a8c270532947ba9b505aace44b56f6 with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:17.178531 15415 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:17.178823 15415 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:17.179548 15415 ts_tablet_manager.cc:1367] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.087s user 0.024s sys 0.004s
I20240503 14:00:17.179988 15415 raft_consensus.cc:348] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:17.180107 15415 raft_consensus.cc:729] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:17.181123 15415 consensus_queue.cc:260] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:17.186275 15415 ts_tablet_manager.cc:1398] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 14:00:17.188532 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-a4b4e16598b740cb9d690c2442a3e994 received from {username='slave'} at 127.0.99.135:46961
I20240503 14:00:17.188607 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-a4b4e16598b740cb9d690c2442a3e994 on tablet a4b4e16598b740cb9d690c2442a3e994 with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:17.198212 14168 raft_consensus.cc:1203] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:17.199986 15540 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:17.201968 15540 tablet_bootstrap.cc:492] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:17.202168 15540 ts_tablet_manager.cc:1367] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.086s user 0.027s sys 0.000s
I20240503 14:00:17.202658 15540 raft_consensus.cc:348] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:17.202786 15540 raft_consensus.cc:729] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:17.202941 15540 consensus_queue.cc:260] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:17.203300 15540 ts_tablet_manager.cc:1398] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:17.203804 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-f5c06806a19e4f4592d26ed4acc651cd received from {username='slave'} at 127.0.99.136:40263
I20240503 14:00:17.203877 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-f5c06806a19e4f4592d26ed4acc651cd on tablet f5c06806a19e4f4592d26ed4acc651cd with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:17.204805 13757 raft_consensus.cc:1203] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:17.218588 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:17.223958 15438 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:17.224251 15438 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:17.224411 15438 ts_tablet_manager.cc:1367] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.363s user 0.025s sys 0.000s
I20240503 14:00:17.224864 15438 raft_consensus.cc:348] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:17.224979 15438 raft_consensus.cc:729] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:17.225109 15438 consensus_queue.cc:260] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:17.225456 15438 ts_tablet_manager.cc:1398] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:17.225953 12450 tablet_copy_service.cc:342] P 84c63c6f551949f6a1d7c07760824361: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-c8933689df2c45b59a9cd16024ddb7d9 received from {username='slave'} at 127.0.99.136:55937
I20240503 14:00:17.226033 12450 tablet_copy_service.cc:433] P 84c63c6f551949f6a1d7c07760824361: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-c8933689df2c45b59a9cd16024ddb7d9 on tablet c8933689df2c45b59a9cd16024ddb7d9 with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:17.228262 15598 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:17.228577 15598 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:17.228781 15598 ts_tablet_manager.cc:1367] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.388s user 0.021s sys 0.005s
I20240503 14:00:17.229249 15598 raft_consensus.cc:348] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:17.229360 15598 raft_consensus.cc:729] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:17.229483 15598 consensus_queue.cc:260] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:17.229786 15598 ts_tablet_manager.cc:1398] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:17.230199 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-3481715d755a4277b0905b0832fad4af received from {username='slave'} at 127.0.99.133:38411
I20240503 14:00:17.230279 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-3481715d755a4277b0905b0832fad4af on tablet 3481715d755a4277b0905b0832fad4af with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:17.231612 14037 raft_consensus.cc:1203] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.231771 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:17.233289 13757 raft_consensus.cc:1203] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.246212 13906 raft_consensus.cc:1203] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.248613 14037 raft_consensus.cc:1203] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:17.248736 12036 consensus_queue.cc:237] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:17.250739 12167 raft_consensus.cc:1261] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:17.250918 12428 raft_consensus.cc:1261] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:17.251121 14896 consensus_queue.cc:1035] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.251353 15485 consensus_queue.cc:1035] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:17.251529 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e442bb664f44887aa8afdb62e1d97fc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.256968 15485 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.257472 12167 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.260012 11895 catalog_manager.cc:5484] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.260901 12428 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.267526 12428 consensus_queue.cc:237] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:17.269129 12036 raft_consensus.cc:1261] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:17.269255 12167 raft_consensus.cc:1261] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:17.269340 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4224ca4d93ce4784b52a7550468ce1b1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.269621 15868 consensus_queue.cc:1035] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.269811 15882 consensus_queue.cc:1035] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.276204 12428 consensus_queue.cc:237] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:17.277438 15869 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.277565 12298 raft_consensus.cc:1261] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:17.278075 12036 raft_consensus.cc:1261] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:17.279271 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70ffc5888a5945d6a8e166420e6f1bd9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.279448 15868 consensus_queue.cc:1035] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.279716 15868 consensus_queue.cc:1035] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.280321 12036 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.282084 12167 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.283362 11892 catalog_manager.cc:5484] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.290242 15882 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.293460 12036 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.295789 12298 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.295931 11895 catalog_manager.cc:5484] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.311662 12298 consensus_queue.cc:237] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:17.313262 12167 raft_consensus.cc:1261] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:00:17.313586 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b031f09b97664d1a94c14f0661181899. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.313752 15858 consensus_queue.cc:1035] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.314566 12036 raft_consensus.cc:1261] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:17.314910 15858 consensus_queue.cc:1035] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.323724 15858 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:17.326555 12167 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:17.327193 14168 raft_consensus.cc:1203] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.328078 12036 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:17.330865 11890 catalog_manager.cc:5484] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:17.338251 14168 raft_consensus.cc:1203] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.347759 12036 consensus_queue.cc:237] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:17.349364 12428 raft_consensus.cc:1261] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:17.349766 14896 consensus_queue.cc:1035] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:17.349828 12167 raft_consensus.cc:1261] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:00:17.350241 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e442bb664f44887aa8afdb62e1d97fc. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:17.350574 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e442bb664f44887aa8afdb62e1d97fc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.350752 14896 consensus_queue.cc:1035] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:17.354183 15485 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:17.357618 12429 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:17.360052 12167 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:17.361620 11889 catalog_manager.cc:5484] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:17.372556 15438 ts_tablet_manager.cc:902] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:17.373044 15438 tablet_copy_client.cc:308] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33927
I20240503 14:00:17.373538 12450 tablet_copy_service.cc:140] P 84c63c6f551949f6a1d7c07760824361: Received BeginTabletCopySession request for tablet 211de6f35a844e278e8c1eb1621e7c5f from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:55937)
I20240503 14:00:17.373652 12450 tablet_copy_service.cc:161] P 84c63c6f551949f6a1d7c07760824361: Beginning new tablet copy session on tablet 211de6f35a844e278e8c1eb1621e7c5f from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:55937: session id = 1b4a61e8a11440e7a9f4b300d2442f75-211de6f35a844e278e8c1eb1621e7c5f
I20240503 14:00:17.374694 12450 tablet_copy_source_session.cc:205] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:17.375759 15438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211de6f35a844e278e8c1eb1621e7c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:17.377823 15438 tablet_copy_client.cc:783] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:17.377975 15438 tablet_copy_client.cc:647] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:17.380730 15438 tablet_copy_client.cc:523] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:17.382076 15438 tablet_bootstrap.cc:492] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:17.392455 12429 consensus_queue.cc:237] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:17.394742 12036 raft_consensus.cc:1261] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:17.394876 12167 raft_consensus.cc:1261] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:00:17.395473 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4224ca4d93ce4784b52a7550468ce1b1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.395642 15869 consensus_queue.cc:1035] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:17.396348 15868 consensus_queue.cc:1035] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:17.399366 15868 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:17.401783 12036 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:17.403321 12167 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:17.404984 11889 catalog_manager.cc:5484] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:17.424422 13757 raft_consensus.cc:1203] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.429037 12036 consensus_queue.cc:237] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:17.431366 12429 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:17.432034 15100 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:17.432722 11972 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 355b5456887b4c5694badc9d53b1809a. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:17.433485 11971 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 355b5456887b4c5694badc9d53b1809a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.435335 12298 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:17.435974 16066 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:17.437379 13906 raft_consensus.cc:1203] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.437916 16066 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
W20240503 14:00:17.440905 12364 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4224ca4d93ce4784b52a7550468ce1b1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.441664 12429 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:17.442310 15438 tablet_bootstrap.cc:492] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:17.442763 15438 tablet_bootstrap.cc:492] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:17.442972 15438 ts_tablet_manager.cc:1367] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.061s user 0.021s sys 0.004s
I20240503 14:00:17.443498 15438 raft_consensus.cc:348] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:17.443609 15438 raft_consensus.cc:729] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:17.443734 15438 consensus_queue.cc:260] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:17.444021 15438 ts_tablet_manager.cc:1398] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:17.444511 12450 tablet_copy_service.cc:342] P 84c63c6f551949f6a1d7c07760824361: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-211de6f35a844e278e8c1eb1621e7c5f received from {username='slave'} at 127.0.99.136:55937
I20240503 14:00:17.444599 12450 tablet_copy_service.cc:433] P 84c63c6f551949f6a1d7c07760824361: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-211de6f35a844e278e8c1eb1621e7c5f on tablet 211de6f35a844e278e8c1eb1621e7c5f with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:17.445861 11886 catalog_manager.cc:5484] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:17.445896 12298 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:17.457067 13906 raft_consensus.cc:1203] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:17.478828 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.487502 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:17.491422 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:17.492121 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.513167 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:17.514025 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.514142 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.515013 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:17.518118 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.521301 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:17.523208 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:17.523336 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.539541 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:17.542438 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.553081 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:17.559865 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.569048 12298 consensus_queue.cc:237] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:17.570864 12036 raft_consensus.cc:1261] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:17.571841 15067 consensus_queue.cc:1035] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:17.574801 14037 raft_consensus.cc:1203] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 14:00:17.575803 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b031f09b97664d1a94c14f0661181899. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.577270 12166 raft_consensus.cc:1261] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:17.577734 15067 consensus_queue.cc:1035] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:17.577843 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b031f09b97664d1a94c14f0661181899. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.579794 15067 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.582368 12036 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.585314 11886 catalog_manager.cc:5484] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 4 to 5, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.587841 12166 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:17.641464 13757 raft_consensus.cc:1203] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.676016 14168 raft_consensus.cc:1203] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:17.692508 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.693486 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.701910 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:17.713593 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:17.727072 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:17.727411 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:17.746034 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.756780 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.818840 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.827646 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:17.843437 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:17.848124 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.852317 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.853649 16303 ts_tablet_manager.cc:902] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:17.854173 16303 tablet_copy_client.cc:308] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33927
I20240503 14:00:17.854598 12450 tablet_copy_service.cc:140] P 84c63c6f551949f6a1d7c07760824361: Received BeginTabletCopySession request for tablet 4224ca4d93ce4784b52a7550468ce1b1 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:43887)
I20240503 14:00:17.854686 12450 tablet_copy_service.cc:161] P 84c63c6f551949f6a1d7c07760824361: Beginning new tablet copy session on tablet 4224ca4d93ce4784b52a7550468ce1b1 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:43887: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-4224ca4d93ce4784b52a7550468ce1b1
I20240503 14:00:17.855023 16345 ts_tablet_manager.cc:902] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:17.855710 12450 tablet_copy_source_session.cc:205] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:17.855856 16345 tablet_copy_client.cc:308] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:17.857386 16303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4224ca4d93ce4784b52a7550468ce1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:17.857388 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet 0e442bb664f44887aa8afdb62e1d97fc from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:60777)
I20240503 14:00:17.857512 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet 0e442bb664f44887aa8afdb62e1d97fc from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:60777: session id = b67f337e10994234baedbf0b19c13513-0e442bb664f44887aa8afdb62e1d97fc
I20240503 14:00:17.858438 12057 tablet_copy_source_session.cc:205] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:17.861910 16345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e442bb664f44887aa8afdb62e1d97fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:17.863577 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:17.865271 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.871539 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.874476 16345 tablet_copy_client.cc:783] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:17.874655 16345 tablet_copy_client.cc:647] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:17.875010 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:17.876520 16303 tablet_copy_client.cc:783] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:17.876686 16303 tablet_copy_client.cc:647] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:17.878513 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.879120 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:17.880088 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:17.884326 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:17.887737 14168 raft_consensus.cc:1203] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:17.917150 16359 ts_tablet_manager.cc:902] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:17.918340 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.919862 16359 tablet_copy_client.cc:308] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33927
I20240503 14:00:17.923631 12450 tablet_copy_service.cc:140] P 84c63c6f551949f6a1d7c07760824361: Received BeginTabletCopySession request for tablet 70ffc5888a5945d6a8e166420e6f1bd9 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:43887)
I20240503 14:00:17.923764 12450 tablet_copy_service.cc:161] P 84c63c6f551949f6a1d7c07760824361: Beginning new tablet copy session on tablet 70ffc5888a5945d6a8e166420e6f1bd9 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:43887: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-70ffc5888a5945d6a8e166420e6f1bd9
I20240503 14:00:17.924490 12450 tablet_copy_source_session.cc:205] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:17.925449 16359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ffc5888a5945d6a8e166420e6f1bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:17.928619 16359 tablet_copy_client.cc:783] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:17.928794 16359 tablet_copy_client.cc:647] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:17.929877 16359 tablet_copy_client.cc:523] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:17.931756 16359 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:17.938187 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:17.952934 16358 ts_tablet_manager.cc:902] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:17.953264 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:17.957988 16303 tablet_copy_client.cc:523] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:17.959470 16303 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:17.963347 16358 tablet_copy_client.cc:308] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:17.963865 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet 0e442bb664f44887aa8afdb62e1d97fc from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:46961)
I20240503 14:00:17.963965 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet 0e442bb664f44887aa8afdb62e1d97fc from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:46961: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-0e442bb664f44887aa8afdb62e1d97fc
I20240503 14:00:17.964743 12298 consensus_queue.cc:237] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:17.964988 12057 tablet_copy_source_session.cc:205] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:17.966133 12429 raft_consensus.cc:1261] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:17.966590 15859 consensus_queue.cc:1035] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:00:17.966924 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df13ec098dc548afac8c67195d25e7e8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:17.966920 12166 raft_consensus.cc:1261] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:17.967300 15859 consensus_queue.cc:1035] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.970057 16358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e442bb664f44887aa8afdb62e1d97fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:17.972452 16358 tablet_copy_client.cc:783] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:17.972656 16358 tablet_copy_client.cc:647] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:17.973161 16345 tablet_copy_client.cc:523] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:17.974511 16345 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:17.976784 16358 tablet_copy_client.cc:523] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:17.977939 16358 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:17.987967 16359 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:17.989967 16303 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:17.990386 16374 ts_tablet_manager.cc:902] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:17.990906 16374 tablet_copy_client.cc:308] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:17.991467 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet 355b5456887b4c5694badc9d53b1809a from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:53301)
I20240503 14:00:17.991567 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet 355b5456887b4c5694badc9d53b1809a from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:53301: session id = 66190085d1c04e068d5c0dec9c4bf07d-355b5456887b4c5694badc9d53b1809a
I20240503 14:00:17.992051 12296 consensus_queue.cc:237] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:17.992478 12057 tablet_copy_source_session.cc:205] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:17.993324 16039 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:17.993813 12166 raft_consensus.cc:1261] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:17.993921 12037 raft_consensus.cc:1261] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:17.994385 16040 consensus_queue.cc:1035] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.994722 16040 consensus_queue.cc:1035] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:17.996150 16374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355b5456887b4c5694badc9d53b1809a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:17.997336 12167 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:17.997440 12429 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:18.000072 11887 catalog_manager.cc:5484] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index -1 to 4, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:18.002295 16374 tablet_copy_client.cc:783] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:18.002503 16374 tablet_copy_client.cc:647] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
W20240503 14:00:18.003993 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adf2c44328e244779955e712b99f8331. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:18.004598 16374 tablet_copy_client.cc:523] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:18.004603 16303 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:18.004840 16303 ts_tablet_manager.cc:1367] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.045s user 0.018s sys 0.005s
I20240503 14:00:18.005297 16303 raft_consensus.cc:348] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:18.005467 16303 raft_consensus.cc:729] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.005770 16359 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:18.005726 16303 consensus_queue.cc:260] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:18.006322 16359 ts_tablet_manager.cc:1367] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.075s user 0.023s sys 0.003s
I20240503 14:00:18.006737 16303 ts_tablet_manager.cc:1398] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:18.007170 12450 tablet_copy_service.cc:342] P 84c63c6f551949f6a1d7c07760824361: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-4224ca4d93ce4784b52a7550468ce1b1 received from {username='slave'} at 127.0.99.135:43887
I20240503 14:00:18.007261 12450 tablet_copy_service.cc:433] P 84c63c6f551949f6a1d7c07760824361: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-4224ca4d93ce4784b52a7550468ce1b1 on tablet 4224ca4d93ce4784b52a7550468ce1b1 with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:18.006916 16392 ts_tablet_manager.cc:902] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:18.007953 16359 raft_consensus.cc:348] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:18.008144 16392 tablet_copy_client.cc:308] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:18.008592 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet 355b5456887b4c5694badc9d53b1809a from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:60777)
I20240503 14:00:18.008723 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet 355b5456887b4c5694badc9d53b1809a from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:60777: session id = b67f337e10994234baedbf0b19c13513-355b5456887b4c5694badc9d53b1809a
I20240503 14:00:18.008106 16359 raft_consensus.cc:729] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.009292 16359 consensus_queue.cc:260] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:18.009554 16359 ts_tablet_manager.cc:1398] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:00:18.009675 12057 tablet_copy_source_session.cc:205] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:18.010005 12450 tablet_copy_service.cc:342] P 84c63c6f551949f6a1d7c07760824361: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-70ffc5888a5945d6a8e166420e6f1bd9 received from {username='slave'} at 127.0.99.135:43887
I20240503 14:00:18.010076 12450 tablet_copy_service.cc:433] P 84c63c6f551949f6a1d7c07760824361: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-70ffc5888a5945d6a8e166420e6f1bd9 on tablet 70ffc5888a5945d6a8e166420e6f1bd9 with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:18.010855 16392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355b5456887b4c5694badc9d53b1809a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.010888 16374 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:18.013235 16392 tablet_copy_client.cc:783] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:18.013381 16392 tablet_copy_client.cc:647] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:18.013931 16390 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:18.018247 12166 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:18.018643 12037 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:18.021265 11895 catalog_manager.cc:5484] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index -1 to 4, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:18.026352 16392 tablet_copy_client.cc:523] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:18.026551 16389 ts_tablet_manager.cc:902] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:18.027089 16389 tablet_copy_client.cc:308] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33927
I20240503 14:00:18.029057 16392 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:18.029836 16345 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:18.030079 16345 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:18.030230 16345 ts_tablet_manager.cc:1367] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.056s user 0.025s sys 0.000s
I20240503 14:00:18.030692 16345 raft_consensus.cc:348] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.030786 16345 raft_consensus.cc:729] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.030896 16345 consensus_queue.cc:260] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.031213 16345 ts_tablet_manager.cc:1398] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 14:00:18.032137 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-0e442bb664f44887aa8afdb62e1d97fc received from {username='slave'} at 127.0.99.134:60777
I20240503 14:00:18.032238 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session b67f337e10994234baedbf0b19c13513-0e442bb664f44887aa8afdb62e1d97fc on tablet 0e442bb664f44887aa8afdb62e1d97fc with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:18.040485 16359 ts_tablet_manager.cc:902] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:18.040921 16359 tablet_copy_client.cc:308] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:18.041795 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet b031f09b97664d1a94c14f0661181899 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:51943)
I20240503 14:00:18.041915 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet b031f09b97664d1a94c14f0661181899 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:51943: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-b031f09b97664d1a94c14f0661181899
I20240503 14:00:18.042825 12319 tablet_copy_source_session.cc:205] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:18.043844 16359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b031f09b97664d1a94c14f0661181899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.045945 16359 tablet_copy_client.cc:783] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:18.046104 16359 tablet_copy_client.cc:647] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:18.055424 16359 tablet_copy_client.cc:523] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:18.056890 16359 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:18.066584 12450 tablet_copy_service.cc:140] P 84c63c6f551949f6a1d7c07760824361: Received BeginTabletCopySession request for tablet 4224ca4d93ce4784b52a7550468ce1b1 from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:53179)
I20240503 14:00:18.066709 12450 tablet_copy_service.cc:161] P 84c63c6f551949f6a1d7c07760824361: Beginning new tablet copy session on tablet 4224ca4d93ce4784b52a7550468ce1b1 from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:53179: session id = 66190085d1c04e068d5c0dec9c4bf07d-4224ca4d93ce4784b52a7550468ce1b1
I20240503 14:00:18.067647 12450 tablet_copy_source_session.cc:205] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:18.067896 16374 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:18.068328 16374 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:18.068560 16374 ts_tablet_manager.cc:1367] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.058s user 0.021s sys 0.005s
I20240503 14:00:18.068758 16389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4224ca4d93ce4784b52a7550468ce1b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.069042 16374 raft_consensus.cc:348] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.069146 16374 raft_consensus.cc:729] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.069274 16374 consensus_queue.cc:260] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.069523 16374 ts_tablet_manager.cc:1398] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:18.070803 16389 tablet_copy_client.cc:783] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:18.070941 16389 tablet_copy_client.cc:647] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:18.071128 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-355b5456887b4c5694badc9d53b1809a received from {username='slave'} at 127.0.99.133:53301
I20240503 14:00:18.071210 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-355b5456887b4c5694badc9d53b1809a on tablet 355b5456887b4c5694badc9d53b1809a with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:18.072710 16389 tablet_copy_client.cc:523] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:18.073894 16389 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:18.093973 12296 consensus_queue.cc:237] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.096297 12037 raft_consensus.cc:1261] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:18.097154 15858 consensus_queue.cc:1035] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:18.098003 12166 raft_consensus.cc:1261] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:18.098140 16359 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:18.098462 16359 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:18.098655 16359 ts_tablet_manager.cc:1367] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.042s user 0.027s sys 0.000s
I20240503 14:00:18.099126 16359 raft_consensus.cc:348] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:18.099267 16359 raft_consensus.cc:729] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.099403 16359 consensus_queue.cc:260] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:18.099711 16359 ts_tablet_manager.cc:1398] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 14:00:18.100973 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adf2c44328e244779955e712b99f8331. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:18.103828 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adf2c44328e244779955e712b99f8331. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:18.104717 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-b031f09b97664d1a94c14f0661181899 received from {username='slave'} at 127.0.99.135:51943
I20240503 14:00:18.104789 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-b031f09b97664d1a94c14f0661181899 on tablet b031f09b97664d1a94c14f0661181899 with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:18.107182 15858 consensus_queue.cc:1035] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:18.110963 15067 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:18.114199 12037 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:18.115132 11892 catalog_manager.cc:5484] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:18.118624 12166 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:18.119431 16392 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:18.119737 16392 tablet_bootstrap.cc:492] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:18.119921 16392 ts_tablet_manager.cc:1367] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.091s user 0.022s sys 0.005s
I20240503 14:00:18.120398 16392 raft_consensus.cc:348] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.120537 16392 raft_consensus.cc:729] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.120661 16392 consensus_queue.cc:260] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.120992 16392 ts_tablet_manager.cc:1398] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:18.121750 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-355b5456887b4c5694badc9d53b1809a received from {username='slave'} at 127.0.99.134:60777
I20240503 14:00:18.121884 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session b67f337e10994234baedbf0b19c13513-355b5456887b4c5694badc9d53b1809a on tablet 355b5456887b4c5694badc9d53b1809a with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:18.131264 16358 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:18.131691 16358 tablet_bootstrap.cc:492] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:18.131922 16358 ts_tablet_manager.cc:1367] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.154s user 0.023s sys 0.004s
I20240503 14:00:18.132407 16358 raft_consensus.cc:348] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.132525 16358 raft_consensus.cc:729] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.132656 16358 consensus_queue.cc:260] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.132946 16358 ts_tablet_manager.cc:1398] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:18.134132 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-0e442bb664f44887aa8afdb62e1d97fc received from {username='slave'} at 127.0.99.135:46961
I20240503 14:00:18.134220 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-0e442bb664f44887aa8afdb62e1d97fc on tablet 0e442bb664f44887aa8afdb62e1d97fc with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:18.141774 16498 ts_tablet_manager.cc:902] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:18.142287 16498 tablet_copy_client.cc:308] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:18.142803 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet b031f09b97664d1a94c14f0661181899 from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:41157)
I20240503 14:00:18.142908 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet b031f09b97664d1a94c14f0661181899 from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:41157: session id = 1b4a61e8a11440e7a9f4b300d2442f75-b031f09b97664d1a94c14f0661181899
I20240503 14:00:18.143870 12319 tablet_copy_source_session.cc:205] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:18.145138 16498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b031f09b97664d1a94c14f0661181899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.150417 16389 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:18.150727 16389 tablet_bootstrap.cc:492] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:18.150916 16389 ts_tablet_manager.cc:1367] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.077s user 0.024s sys 0.000s
I20240503 14:00:18.151628 16498 tablet_copy_client.cc:783] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:18.160981 16498 tablet_copy_client.cc:647] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:18.161523 16389 raft_consensus.cc:348] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:18.161733 16389 raft_consensus.cc:729] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.161934 16389 consensus_queue.cc:260] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:18.162925 16389 ts_tablet_manager.cc:1398] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:18.163396 12450 tablet_copy_service.cc:342] P 84c63c6f551949f6a1d7c07760824361: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-4224ca4d93ce4784b52a7550468ce1b1 received from {username='slave'} at 127.0.99.133:53179
I20240503 14:00:18.163468 12450 tablet_copy_service.cc:433] P 84c63c6f551949f6a1d7c07760824361: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-4224ca4d93ce4784b52a7550468ce1b1 on tablet 4224ca4d93ce4784b52a7550468ce1b1 with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:18.171892 16498 tablet_copy_client.cc:523] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:18.173735 16498 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:18.199069 16498 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:18.201023 16498 tablet_bootstrap.cc:492] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:18.201845 16498 ts_tablet_manager.cc:1367] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.000s
I20240503 14:00:18.202425 16498 raft_consensus.cc:348] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:18.202571 16498 raft_consensus.cc:729] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.202731 16498 consensus_queue.cc:260] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:18.203670 16498 ts_tablet_manager.cc:1398] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:18.204051 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-b031f09b97664d1a94c14f0661181899 received from {username='slave'} at 127.0.99.136:41157
I20240503 14:00:18.204128 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-b031f09b97664d1a94c14f0661181899 on tablet b031f09b97664d1a94c14f0661181899 with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:18.208353 12296 consensus_queue.cc:237] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:18.209834 12166 raft_consensus.cc:1261] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:18.209864 12429 raft_consensus.cc:1261] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:00:18.210147 12233 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df13ec098dc548afac8c67195d25e7e8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:18.210229 15859 consensus_queue.cc:1035] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:18.210336 15067 consensus_queue.cc:1035] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:00:18.210569 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df13ec098dc548afac8c67195d25e7e8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:18.212558 15067 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:18.214645 12429 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:18.216396 12166 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:18.217075 11893 catalog_manager.cc:5484] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 4 to 5, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:18.224529 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:18.225033 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:18.228307 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:18.229666 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.234820 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.249192 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.253682 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.273135 14037 raft_consensus.cc:1203] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:00:18.284142 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:18.305088 14037 raft_consensus.cc:1203] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:18.315845 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.317103 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.318941 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:18.327633 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.328173 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:18.329089 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:18.347792 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.369244 13757 raft_consensus.cc:1203] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:18.386293 13757 raft_consensus.cc:1203] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:18.395396 13906 raft_consensus.cc:1203] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:18.398267 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:18.457577 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.458045 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.480051 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:18.487778 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.487810 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:18.489912 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:18.490257 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:18.491636 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:18.497345 14037 raft_consensus.cc:1203] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:18.506705 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.509292 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.518314 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.532181 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:18.540275 13906 raft_consensus.cc:1203] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:18.546065 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.548655 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:18.550851 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.566953 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:18.584192 14037 raft_consensus.cc:1203] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:18.602872 16358 ts_tablet_manager.cc:902] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:18.603368 16358 tablet_copy_client.cc:308] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:18.603960 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet adf2c44328e244779955e712b99f8331 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:51943)
I20240503 14:00:18.604063 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet adf2c44328e244779955e712b99f8331 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:51943: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-adf2c44328e244779955e712b99f8331
I20240503 14:00:18.604921 12319 tablet_copy_source_session.cc:205] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:18.605762 16358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf2c44328e244779955e712b99f8331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.607954 16358 tablet_copy_client.cc:783] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:18.608165 16358 tablet_copy_client.cc:647] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:18.610273 16358 tablet_copy_client.cc:523] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:18.611939 16358 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:18.624953 16498 ts_tablet_manager.cc:902] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:18.625419 16498 tablet_copy_client.cc:308] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:18.626035 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet df13ec098dc548afac8c67195d25e7e8 from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:41157)
I20240503 14:00:18.626130 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet df13ec098dc548afac8c67195d25e7e8 from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:41157: session id = 1b4a61e8a11440e7a9f4b300d2442f75-df13ec098dc548afac8c67195d25e7e8
I20240503 14:00:18.627034 12319 tablet_copy_source_session.cc:205] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:18.627961 16498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df13ec098dc548afac8c67195d25e7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.630187 16498 tablet_copy_client.cc:783] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:18.630327 16498 tablet_copy_client.cc:647] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:18.632575 16498 tablet_copy_client.cc:523] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:18.633810 16498 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:18.643623 16358 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:18.643908 16358 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:18.644085 16358 ts_tablet_manager.cc:1367] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.003s
I20240503 14:00:18.644503 16358 raft_consensus.cc:348] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.644632 16358 raft_consensus.cc:729] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.644779 16358 consensus_queue.cc:260] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.645124 16358 ts_tablet_manager.cc:1398] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:18.645707 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-adf2c44328e244779955e712b99f8331 received from {username='slave'} at 127.0.99.135:51943
I20240503 14:00:18.645792 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-adf2c44328e244779955e712b99f8331 on tablet adf2c44328e244779955e712b99f8331 with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:18.658514 14167 raft_consensus.cc:1203] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:18.671797 16498 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:18.672129 16498 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:18.672297 16498 ts_tablet_manager.cc:1367] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.039s user 0.023s sys 0.000s
I20240503 14:00:18.677055 16498 raft_consensus.cc:348] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:18.678130 16498 raft_consensus.cc:729] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.678468 16498 consensus_queue.cc:260] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:18.681432 16498 ts_tablet_manager.cc:1398] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:00:18.682145 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-df13ec098dc548afac8c67195d25e7e8 received from {username='slave'} at 127.0.99.136:41157
I20240503 14:00:18.682225 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-df13ec098dc548afac8c67195d25e7e8 on tablet df13ec098dc548afac8c67195d25e7e8 with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:18.698148 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:18.700956 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.705996 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.709532 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:18.722978 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.724895 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:18.725164 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.750816 16796 ts_tablet_manager.cc:902] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:18.751392 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:18.751575 16796 tablet_copy_client.cc:308] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:18.775408 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet adf2c44328e244779955e712b99f8331 from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:41709)
I20240503 14:00:18.775468 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.775538 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet adf2c44328e244779955e712b99f8331 from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:41709: session id = b67f337e10994234baedbf0b19c13513-adf2c44328e244779955e712b99f8331
I20240503 14:00:18.776521 12319 tablet_copy_source_session.cc:205] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:18.778903 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.780718 16796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf2c44328e244779955e712b99f8331. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.783318 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:18.786269 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:18.795112 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.797044 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.799445 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:18.810230 16796 tablet_copy_client.cc:783] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:18.810468 16796 tablet_copy_client.cc:647] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:18.818130 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:18.822279 16796 tablet_copy_client.cc:523] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:18.823871 16796 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:18.846027 16842 ts_tablet_manager.cc:902] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:18.846592 16842 tablet_copy_client.cc:308] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:18.847052 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet df13ec098dc548afac8c67195d25e7e8 from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:38411)
I20240503 14:00:18.847153 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet df13ec098dc548afac8c67195d25e7e8 from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:38411: session id = 66190085d1c04e068d5c0dec9c4bf07d-df13ec098dc548afac8c67195d25e7e8
I20240503 14:00:18.848074 12319 tablet_copy_source_session.cc:205] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:18.850026 16796 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:18.850358 16796 tablet_bootstrap.cc:492] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:18.850544 16796 ts_tablet_manager.cc:1367] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.027s user 0.020s sys 0.003s
I20240503 14:00:18.851078 16796 raft_consensus.cc:348] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.851199 16796 raft_consensus.cc:729] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.851375 16796 consensus_queue.cc:260] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:18.851639 16842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df13ec098dc548afac8c67195d25e7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:18.852490 16796 ts_tablet_manager.cc:1398] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240503 14:00:18.852919 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-adf2c44328e244779955e712b99f8331 received from {username='slave'} at 127.0.99.134:41709
I20240503 14:00:18.853001 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session b67f337e10994234baedbf0b19c13513-adf2c44328e244779955e712b99f8331 on tablet adf2c44328e244779955e712b99f8331 with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:18.854461 16842 tablet_copy_client.cc:783] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:18.854671 16842 tablet_copy_client.cc:647] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:18.859849 16842 tablet_copy_client.cc:523] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:18.861253 16842 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:18.892800 16842 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:18.893122 16842 tablet_bootstrap.cc:492] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:18.893357 16842 ts_tablet_manager.cc:1367] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.000s
I20240503 14:00:18.893885 16842 raft_consensus.cc:348] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:18.894016 16842 raft_consensus.cc:729] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:18.894157 16842 consensus_queue.cc:260] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:18.894579 16842 ts_tablet_manager.cc:1398] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:18.895905 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-df13ec098dc548afac8c67195d25e7e8 received from {username='slave'} at 127.0.99.133:38411
I20240503 14:00:18.895998 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-df13ec098dc548afac8c67195d25e7e8 on tablet df13ec098dc548afac8c67195d25e7e8 with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:18.934909 12037 consensus_queue.cc:237] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:18.936600 13757 raft_consensus.cc:1261] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:18.937175 12429 raft_consensus.cc:1261] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:18.937971 14896 consensus_queue.cc:1035] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:18.938151 16065 consensus_queue.cc:1035] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:18.949738 16065 raft_consensus.cc:2932] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:18.956499 12429 raft_consensus.cc:2932] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:18.957302 11886 catalog_manager.cc:5484] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:18.958384 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:18.959551 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.971045 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:18.977404 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:18.990727 13775 raft_consensus.cc:2932] T 4a608f683c4f41f3bbdfc9b4c219ce49 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:18.995738 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.012162 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.023897 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:19.032294 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.048491 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.054432 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:19.056375 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.075361 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.078925 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:19.079041 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.080029 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:19.082398 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.087990 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:19.096518 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:19.102722 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.105031 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:19.127110 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.138257 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.138520 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a608f683c4f41f3bbdfc9b4c219ce49 with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:19.144611 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.157706 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.169858 17023 tablet_replica.cc:330] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:19.171061 17023 raft_consensus.cc:2229] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:19.174247 17023 raft_consensus.cc:2258] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:19.176301 17023 ts_tablet_manager.cc:1875] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:19.192209 17023 ts_tablet_manager.cc:1888] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:19.195211 17023 log.cc:1196] T 4a608f683c4f41f3bbdfc9b4c219ce49 P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4a608f683c4f41f3bbdfc9b4c219ce49
I20240503 14:00:19.195976 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet 4a608f683c4f41f3bbdfc9b4c219ce49 (table rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]) successfully deleted
I20240503 14:00:19.203876 12016 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:19.204022 14167 raft_consensus.cc:1203] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:19.210676 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.211827 13755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.214872 13757 raft_consensus.cc:1203] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:19.215950 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.218246 14037 raft_consensus.cc:1203] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:19.228324 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:19.228716 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:19.243650 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.246115 12037 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e6ca408c4254678b67a74f49a34881b"
dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39474
I20240503 14:00:19.246315 12037 raft_consensus.cc:595] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:19.253365 14896 raft_consensus.cc:979] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: : Instructing follower 66190085d1c04e068d5c0dec9c4bf07d to start an election
I20240503 14:00:19.253470 14896 raft_consensus.cc:1067] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Signalling peer 66190085d1c04e068d5c0dec9c4bf07d to start an election
I20240503 14:00:19.254127 13757 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e6ca408c4254678b67a74f49a34881b"
dest_uuid: "66190085d1c04e068d5c0dec9c4bf07d"
from {username='slave'} at 127.0.99.129:52947
I20240503 14:00:19.254297 13757 raft_consensus.cc:482] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:19.254390 13757 raft_consensus.cc:3037] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:19.255633 13757 raft_consensus.cc:504] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:19.256831 13757 leader_election.cc:288] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [CANDIDATE]: Term 2 election: Requested vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:19.257861 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6ca408c4254678b67a74f49a34881b" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:19.257982 12296 raft_consensus.cc:3037] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:19.258993 12296 raft_consensus.cc:2454] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:19.259794 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6ca408c4254678b67a74f49a34881b" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:19.259910 12037 raft_consensus.cc:3032] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:19.259955 12037 raft_consensus.cc:729] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:19.260116 12037 consensus_queue.cc:260] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:19.260268 12037 raft_consensus.cc:3037] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:19.261161 12037 raft_consensus.cc:2454] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:19.265468 13711 leader_election.cc:302] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 66190085d1c04e068d5c0dec9c4bf07d, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:19.265724 16892 raft_consensus.cc:2781] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:19.265899 16892 raft_consensus.cc:686] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Becoming Leader. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Running, Role: LEADER
I20240503 14:00:19.266163 16892 consensus_queue.cc:237] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:19.268059 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.268667 11895 catalog_manager.cc:5484] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: term changed from 1 to 2, leader changed from 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) to 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133). New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:19.291065 12166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6ca408c4254678b67a74f49a34881b" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:19.291083 12037 consensus_queue.cc:237] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:19.291590 12166 raft_consensus.cc:3037] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:19.292551 13757 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:19.292695 12166 raft_consensus.cc:2454] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:19.292848 12167 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:19.293020 16953 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:19.294169 16065 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:19.295580 16065 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:19.297304 12167 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:19.298765 13757 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:19.302812 11895 catalog_manager.cc:5484] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:19.304191 13906 raft_consensus.cc:1203] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:00:19.309396 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet faa0a6ca00d442eaaf9fca7fc1259ec7 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:19.314667 17084 tablet_replica.cc:330] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:19.315217 17084 raft_consensus.cc:2229] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:19.315529 17084 raft_consensus.cc:2258] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:19.316687 17084 ts_tablet_manager.cc:1875] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:19.320969 17084 ts_tablet_manager.cc:1888] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:19.321069 17084 log.cc:1196] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/faa0a6ca00d442eaaf9fca7fc1259ec7
I20240503 14:00:19.321906 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet faa0a6ca00d442eaaf9fca7fc1259ec7 (table rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]) successfully deleted
I20240503 14:00:19.375602 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:19.377590 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:19.377851 12016 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.397938 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.400226 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.407292 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:19.425704 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.438582 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.438773 12037 consensus_queue.cc:237] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:19.440982 13757 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:19.441434 14896 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:19.444430 14896 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:19.446525 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:19.447721 13757 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:19.448484 11887 catalog_manager.cc:5484] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:19.448782 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:19.452450 12037 consensus_queue.cc:237] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:19.453191 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet faa0a6ca00d442eaaf9fca7fc1259ec7 with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33028
W20240503 14:00:19.455596 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faa0a6ca00d442eaaf9fca7fc1259ec7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:19.455983 13757 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:19.456346 16953 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:19.457851 16953 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:19.459879 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet faa0a6ca00d442eaaf9fca7fc1259ec7 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:19.461045 13757 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:19.463716 11886 catalog_manager.cc:5484] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: config changed from index 6 to 7, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:19.464654 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.481024 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.496206 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.508000 17203 tablet_replica.cc:330] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143: stopping tablet replica
I20240503 14:00:19.508544 17203 raft_consensus.cc:2229] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:19.508827 17203 raft_consensus.cc:2258] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:19.510337 17203 ts_tablet_manager.cc:1875] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:19.513397 17203 ts_tablet_manager.cc:1888] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:19.513561 17203 log.cc:1196] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 8b17d706d2344dbaae13cda9b0733143: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/faa0a6ca00d442eaaf9fca7fc1259ec7
I20240503 14:00:19.514263 11882 catalog_manager.cc:4830] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): tablet faa0a6ca00d442eaaf9fca7fc1259ec7 (table rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]) successfully deleted
I20240503 14:00:19.525843 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:19.528367 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.540473 12167 consensus_queue.cc:237] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:19.542063 12016 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.542155 14036 raft_consensus.cc:1261] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:19.542032 12037 raft_consensus.cc:1261] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:19.542778 15579 consensus_queue.cc:1035] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:19.543119 15579 consensus_queue.cc:1035] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:19.557184 15579 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:19.562500 14036 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:19.562882 11886 catalog_manager.cc:5484] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:19.565321 12037 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:19.569358 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f9cb87c70e43dbb0d850c275ebb8a5 with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:19.569623 17023 tablet_replica.cc:330] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:19.569770 17023 raft_consensus.cc:2229] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:19.569908 17023 raft_consensus.cc:2258] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:19.570420 17023 ts_tablet_manager.cc:1875] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:19.572631 17023 ts_tablet_manager.cc:1888] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:19.572697 17023 log.cc:1196] T f4f9cb87c70e43dbb0d850c275ebb8a5 P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f4f9cb87c70e43dbb0d850c275ebb8a5
I20240503 14:00:19.573501 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet f4f9cb87c70e43dbb0d850c275ebb8a5 (table rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]) successfully deleted
I20240503 14:00:19.576058 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.592366 12016 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.603435 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.603789 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:19.605216 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:19.606272 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:19.611390 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.627607 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.648972 12167 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:19.649582 17064 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:19.670195 12298 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:19.670680 17064 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:19.678512 12037 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:19.680977 16892 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:19.697746 12037 consensus_queue.cc:237] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:19.699452 14036 raft_consensus.cc:1261] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:19.699828 12167 raft_consensus.cc:1261] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:19.700479 14896 consensus_queue.cc:1035] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:19.700796 14896 consensus_queue.cc:1035] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:19.710858 16065 raft_consensus.cc:2932] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:19.713201 14036 raft_consensus.cc:2932] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:19.716531 11889 catalog_manager.cc:5484] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5 reported cstate change: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:19.719448 12167 raft_consensus.cc:2932] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:19.722532 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet d837845c6f644bca83315c9f7c45246d with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:19.723198 17023 tablet_replica.cc:330] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:19.723359 17023 raft_consensus.cc:2229] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:19.723517 17023 raft_consensus.cc:2258] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:19.724076 17023 ts_tablet_manager.cc:1875] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:19.726290 17023 ts_tablet_manager.cc:1888] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:19.726351 17023 log.cc:1196] T d837845c6f644bca83315c9f7c45246d P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d837845c6f644bca83315c9f7c45246d
I20240503 14:00:19.728336 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet d837845c6f644bca83315c9f7c45246d (table rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]) successfully deleted
I20240503 14:00:19.731922 12298 consensus_queue.cc:237] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:19.733544 14167 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:19.734102 15859 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:19.734925 12429 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:19.736891 15859 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:19.740208 15859 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:19.740497 12429 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:19.743353 11886 catalog_manager.cc:5484] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:19.746918 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet a6a89230883347d4a64d96d321247012 with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33028
I20240503 14:00:19.747130 17203 tablet_replica.cc:330] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143: stopping tablet replica
I20240503 14:00:19.747280 17203 raft_consensus.cc:2229] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:19.747400 17203 raft_consensus.cc:2258] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:19.747149 14167 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:19.747962 17203 ts_tablet_manager.cc:1875] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:19.750242 17203 ts_tablet_manager.cc:1888] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:19.750303 17203 log.cc:1196] T a6a89230883347d4a64d96d321247012 P 8b17d706d2344dbaae13cda9b0733143: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a6a89230883347d4a64d96d321247012
I20240503 14:00:19.752755 11882 catalog_manager.cc:4830] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): tablet a6a89230883347d4a64d96d321247012 (table rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]) successfully deleted
I20240503 14:00:19.805217 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.810083 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:19.811954 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.818059 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.837215 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:19.840420 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.915939 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:19.932572 12016 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.961194 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:19.962394 12016 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:19.970517 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.979363 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:19.989195 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.017874 12016 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.022873 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:20.025842 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.028367 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.038604 17471 ts_tablet_manager.cc:902] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:20.039180 17471 tablet_copy_client.cc:308] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:20.040267 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet faa0a6ca00d442eaaf9fca7fc1259ec7 from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:60777)
I20240503 14:00:20.040365 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet faa0a6ca00d442eaaf9fca7fc1259ec7 from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:60777: session id = b67f337e10994234baedbf0b19c13513-faa0a6ca00d442eaaf9fca7fc1259ec7
I20240503 14:00:20.041275 12057 tablet_copy_source_session.cc:205] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:20.041687 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.050643 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.066447 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.066679 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:20.071135 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.073612 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:20.077422 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.078446 12016 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.080850 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.088379 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.097543 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.122056 17471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa0a6ca00d442eaaf9fca7fc1259ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:20.126096 17471 tablet_copy_client.cc:783] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:20.138113 17471 tablet_copy_client.cc:647] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:20.139371 17471 tablet_copy_client.cc:523] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:20.143718 17471 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:20.147017 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.155331 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.157507 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.160315 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.215049 12037 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd"
dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43536
I20240503 14:00:20.215219 12037 raft_consensus.cc:595] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:20.221359 17471 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:20.221705 17471 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:20.221891 17471 ts_tablet_manager.cc:1367] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.078s user 0.024s sys 0.004s
I20240503 14:00:20.222388 17471 raft_consensus.cc:348] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:20.222512 17471 raft_consensus.cc:729] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:20.222666 17471 consensus_queue.cc:260] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:20.223053 17471 ts_tablet_manager.cc:1398] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:20.224202 12037 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496"
dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43538
I20240503 14:00:20.224326 12037 raft_consensus.cc:595] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:20.225553 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-faa0a6ca00d442eaaf9fca7fc1259ec7 received from {username='slave'} at 127.0.99.134:60777
I20240503 14:00:20.225625 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session b67f337e10994234baedbf0b19c13513-faa0a6ca00d442eaaf9fca7fc1259ec7 on tablet faa0a6ca00d442eaaf9fca7fc1259ec7 with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:20.227797 12296 consensus_queue.cc:237] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:20.229080 14167 raft_consensus.cc:1261] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:20.229179 12429 raft_consensus.cc:1261] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:20.229494 15859 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:20.229810 15859 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:20.238611 12016 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.239272 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.239918 16390 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:20.243988 12428 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:20.244518 17314 raft_consensus.cc:979] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: : Instructing follower 84c63c6f551949f6a1d7c07760824361 to start an election
I20240503 14:00:20.244601 17314 raft_consensus.cc:1067] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Signalling peer 84c63c6f551949f6a1d7c07760824361 to start an election
I20240503 14:00:20.245013 12429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496"
dest_uuid: "84c63c6f551949f6a1d7c07760824361"
from {username='slave'} at 127.0.99.129:58401
I20240503 14:00:20.245123 12429 raft_consensus.cc:482] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:20.245162 12429 raft_consensus.cc:3037] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:20.245896 12429 raft_consensus.cc:504] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:20.246366 12429 leader_election.cc:288] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 2 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075)
I20240503 14:00:20.247676 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:20.247815 12167 raft_consensus.cc:3037] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:20.248399 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:20.248523 12037 raft_consensus.cc:3032] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:20.248574 12037 raft_consensus.cc:729] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:20.248724 12037 consensus_queue.cc:260] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:20.248912 12037 raft_consensus.cc:3037] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:20.248932 12167 raft_consensus.cc:2454] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
I20240503 14:00:20.249969 12037 raft_consensus.cc:2454] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
I20240503 14:00:20.250316 12366 leader_election.cc:302] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:20.250483 13682 raft_consensus.cc:2781] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:20.250564 13682 raft_consensus.cc:686] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 2 LEADER]: Becoming Leader. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Running, Role: LEADER
I20240503 14:00:20.250738 13682 consensus_queue.cc:237] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:20.252009 11887 catalog_manager.cc:5484] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:20.252946 14167 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:20.254904 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:20.258327 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.268749 14017 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.280304 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.291627 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.322095 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.330260 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet c2e9f6eecbec45988b290775277dbddc with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33028
I20240503 14:00:20.331423 11893 catalog_manager.cc:5484] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: term changed from 1 to 2, leader changed from 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) to 84c63c6f551949f6a1d7c07760824361 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:20.337999 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b67f337e10994234baedbf0b19c13513"
I20240503 14:00:20.338225 13906 raft_consensus.cc:3037] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:20.339455 13906 raft_consensus.cc:2454] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
I20240503 14:00:20.344828 12037 consensus_queue.cc:237] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:20.346839 14036 raft_consensus.cc:1261] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:20.347226 12167 raft_consensus.cc:1261] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:20.348748 13906 raft_consensus.cc:1261] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:20.349951 14896 consensus_queue.cc:1035] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:20.350291 14896 consensus_queue.cc:1035] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:20.350909 16953 consensus_queue.cc:1035] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:20.354593 17620 tablet_replica.cc:330] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143: stopping tablet replica
I20240503 14:00:20.354830 17620 raft_consensus.cc:2229] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:20.355088 17620 raft_consensus.cc:2258] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:20.355882 17620 ts_tablet_manager.cc:1875] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:20.358584 17620 ts_tablet_manager.cc:1888] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:20.358727 17620 log.cc:1196] T c2e9f6eecbec45988b290775277dbddc P 8b17d706d2344dbaae13cda9b0733143: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c2e9f6eecbec45988b290775277dbddc
I20240503 14:00:20.359361 11882 catalog_manager.cc:4830] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): tablet c2e9f6eecbec45988b290775277dbddc (table rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]) successfully deleted
I20240503 14:00:20.361937 16953 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:20.364732 13906 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:20.365805 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.366695 14036 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:20.371173 11895 catalog_manager.cc:5484] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513 reported cstate change: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:20.374604 17314 raft_consensus.cc:979] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: : Instructing follower e85b0283a0fd4e3ba81b4170f3cdacb5 to start an election
I20240503 14:00:20.374713 17314 raft_consensus.cc:1067] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Signalling peer e85b0283a0fd4e3ba81b4170f3cdacb5 to start an election
I20240503 14:00:20.375571 14036 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd"
dest_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5"
from {username='slave'} at 127.0.99.129:45763
I20240503 14:00:20.375744 14036 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:20.375839 14036 raft_consensus.cc:3037] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:20.376885 14036 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:20.388896 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:20.389565 12296 raft_consensus.cc:3037] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:20.390862 12296 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85b0283a0fd4e3ba81b4170f3cdacb5 in term 2.
I20240503 14:00:20.392923 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:20.393059 12037 raft_consensus.cc:3032] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:20.393102 12037 raft_consensus.cc:729] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:20.393242 12037 consensus_queue.cc:260] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:20.393472 12037 raft_consensus.cc:3037] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:20.394560 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85b0283a0fd4e3ba81b4170f3cdacb5 in term 2.
I20240503 14:00:20.395081 13973 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, ca5386f84994496fa5376624391728ee, e85b0283a0fd4e3ba81b4170f3cdacb5; no voters:
I20240503 14:00:20.395421 17638 raft_consensus.cc:2781] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:20.395625 17638 raft_consensus.cc:686] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 2 LEADER]: Becoming Leader. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Running, Role: LEADER
I20240503 14:00:20.395973 17638 consensus_queue.cc:237] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:20.398140 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:20.399471 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:20.399596 12429 raft_consensus.cc:3037] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:20.400667 12429 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85b0283a0fd4e3ba81b4170f3cdacb5 in term 2.
I20240503 14:00:20.401235 14036 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [CANDIDATE]: Term 2 election: Requested vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763)
I20240503 14:00:20.405977 12166 consensus_queue.cc:237] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:20.406219 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e442bb664f44887aa8afdb62e1d97fc with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:20.406903 17678 tablet_replica.cc:330] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:20.407004 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1b4a61e8a11440e7a9f4b300d2442f75"
I20240503 14:00:20.407107 17678 raft_consensus.cc:2229] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:20.407130 11893 catalog_manager.cc:5484] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 reported cstate change: term changed from 1 to 2, leader changed from 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) to e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:20.407290 17678 raft_consensus.cc:2258] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:20.407198 14168 raft_consensus.cc:3037] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:20.407874 17678 ts_tablet_manager.cc:1875] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:20.408591 14168 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85b0283a0fd4e3ba81b4170f3cdacb5 in term 2.
I20240503 14:00:20.408982 12037 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:20.409513 17234 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:20.409716 13906 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:20.410133 17234 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:20.410300 17678 ts_tablet_manager.cc:1888] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:20.410367 17678 log.cc:1196] T 0e442bb664f44887aa8afdb62e1d97fc P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0e442bb664f44887aa8afdb62e1d97fc
I20240503 14:00:20.411046 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet 0e442bb664f44887aa8afdb62e1d97fc (table rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]) successfully deleted
I20240503 14:00:20.413077 17234 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:20.416425 11893 catalog_manager.cc:5484] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:20.419588 12167 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:20.421972 12037 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:20.422142 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet c42fee9fadbb4601a7fe32eb6a04e767 with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:20.426492 13906 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:20.430872 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.441246 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.451272 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.454706 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.475013 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.479940 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:20.497491 17692 tablet_replica.cc:330] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:20.497804 17692 raft_consensus.cc:2229] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:20.498006 17692 raft_consensus.cc:2258] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:20.501034 17692 ts_tablet_manager.cc:1875] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:20.521929 17692 ts_tablet_manager.cc:1888] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:20.522173 17692 log.cc:1196] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c42fee9fadbb4601a7fe32eb6a04e767
I20240503 14:00:20.527580 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet c42fee9fadbb4601a7fe32eb6a04e767 (table rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]) successfully deleted
I20240503 14:00:20.538163 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:20.539736 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.549324 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.562431 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.571092 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.572947 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.587179 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:20.589287 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.611763 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:20.617130 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:20.621620 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.628903 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.641723 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.643103 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.645666 13906 raft_consensus.cc:1203] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:00:20.655637 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.658460 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.721477 13906 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:20.722892 13682 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:20.745081 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.749133 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:20.753533 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:20.754048 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.760452 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.765730 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.777824 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.780694 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.782064 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:20.785822 12428 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer e85b0283a0fd4e3ba81b4170f3cdacb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:20.786331 17654 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:20.820078 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.824712 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:20.832669 12037 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:20.833134 13682 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:20.840281 12036 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Refusing update from remote peer e85b0283a0fd4e3ba81b4170f3cdacb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:20.840741 17654 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:20.844240 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.845670 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:20.855818 12167 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:00:20.858198 13682 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:20.862774 14168 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Refusing update from remote peer e85b0283a0fd4e3ba81b4170f3cdacb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:20.863377 17654 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:20.863584 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.864187 12296 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Refusing update from remote peer e85b0283a0fd4e3ba81b4170f3cdacb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:20.864876 17654 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:20.872313 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:20.880633 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.911371 12296 consensus_queue.cc:237] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:20.912753 14168 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:20.913195 12167 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:20.913733 16390 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:20.914942 16390 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:20.918591 16390 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:20.920919 14168 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:20.924691 11894 catalog_manager.cc:5484] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:20.928951 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet a0106f7a40cb467eb4738890ac4e22d4 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:20.929302 17933 tablet_replica.cc:330] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:20.929922 12167 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:20.932112 17933 raft_consensus.cc:2229] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:20.932381 17933 raft_consensus.cc:2258] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:20.933455 17933 ts_tablet_manager.cc:1875] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:20.935758 17933 ts_tablet_manager.cc:1888] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:20.935853 17933 log.cc:1196] T a0106f7a40cb467eb4738890ac4e22d4 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a0106f7a40cb467eb4738890ac4e22d4
I20240503 14:00:20.936393 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet a0106f7a40cb467eb4738890ac4e22d4 (table rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]) successfully deleted
I20240503 14:00:20.966032 12428 consensus_queue.cc:237] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:20.968194 14036 raft_consensus.cc:1261] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:20.968192 12036 raft_consensus.cc:1261] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:20.969623 17872 consensus_queue.cc:1035] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:20.970021 17872 consensus_queue.cc:1035] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:20.970417 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:20.971061 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:20.972081 13757 raft_consensus.cc:1261] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:20.972478 17868 consensus_queue.cc:1035] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:20.986685 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.986912 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.988287 17868 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:20.989595 13757 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:20.990607 12036 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:20.994817 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:20.995433 14037 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:20.997350 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:20.997866 11894 catalog_manager.cc:5484] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:21.006807 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:21.007328 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet 4224ca4d93ce4784b52a7550468ce1b1 with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33028
I20240503 14:00:21.013217 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:21.022828 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.024253 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.030622 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:21.037101 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.037787 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:21.038002 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:21.057091 18001 tablet_replica.cc:330] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143: stopping tablet replica
I20240503 14:00:21.057276 18001 raft_consensus.cc:2229] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:21.057478 18001 raft_consensus.cc:2258] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:21.058107 18001 ts_tablet_manager.cc:1875] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:21.061105 18001 ts_tablet_manager.cc:1888] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:21.061239 18001 log.cc:1196] T 4224ca4d93ce4784b52a7550468ce1b1 P 8b17d706d2344dbaae13cda9b0733143: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4224ca4d93ce4784b52a7550468ce1b1
I20240503 14:00:21.065976 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:21.066004 11882 catalog_manager.cc:4830] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): tablet 4224ca4d93ce4784b52a7550468ce1b1 (table rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]) successfully deleted
I20240503 14:00:21.096172 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.186806 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.188212 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:21.188218 12296 consensus_queue.cc:237] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:21.190107 13757 raft_consensus.cc:1261] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:21.190629 17566 consensus_queue.cc:1035] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:21.190919 12428 raft_consensus.cc:1261] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:21.191370 17566 consensus_queue.cc:1035] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:21.195981 17566 raft_consensus.cc:2932] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:21.196652 12428 raft_consensus.cc:2932] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:21.198259 13757 raft_consensus.cc:2932] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:21.199010 11895 catalog_manager.cc:5484] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:21.199798 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.201653 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet 3481715d755a4277b0905b0832fad4af with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33028
I20240503 14:00:21.201901 18001 tablet_replica.cc:330] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143: stopping tablet replica
I20240503 14:00:21.202034 18001 raft_consensus.cc:2229] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:21.202172 18001 raft_consensus.cc:2258] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:21.202777 18001 ts_tablet_manager.cc:1875] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:21.205708 18001 ts_tablet_manager.cc:1888] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:21.205775 18001 log.cc:1196] T 3481715d755a4277b0905b0832fad4af P 8b17d706d2344dbaae13cda9b0733143: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3481715d755a4277b0905b0832fad4af
I20240503 14:00:21.206266 11882 catalog_manager.cc:4830] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): tablet 3481715d755a4277b0905b0832fad4af (table rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]) successfully deleted
I20240503 14:00:21.209829 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:21.225613 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:21.227516 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.234758 12296 consensus_queue.cc:237] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:21.236732 13757 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:21.237452 16390 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:21.239140 12036 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:21.243541 16390 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:21.244074 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.254907 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:21.264163 16390 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:21.264536 12036 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:21.267973 11895 catalog_manager.cc:5484] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:21.268841 13775 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:21.272771 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet 92506cde7b084b14b88d4a915731eeb3 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:21.273684 17933 tablet_replica.cc:330] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:21.273833 17933 raft_consensus.cc:2229] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:21.274015 17933 raft_consensus.cc:2258] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:21.274677 17933 ts_tablet_manager.cc:1875] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:21.276458 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.276849 17933 ts_tablet_manager.cc:1888] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:21.276908 17933 log.cc:1196] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/92506cde7b084b14b88d4a915731eeb3
I20240503 14:00:21.277449 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet 92506cde7b084b14b88d4a915731eeb3 (table rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]) successfully deleted
I20240503 14:00:21.289992 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:21.305089 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.308930 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.312564 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.323647 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:21.333078 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:21.333338 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:21.401037 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.407423 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:21.411839 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:21.415740 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.437112 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.438671 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:21.451150 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.459003 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:21.533440 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.549425 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.555117 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:21.556015 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:21.560072 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.560328 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:21.563160 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.565369 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:21.586294 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:21.596702 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.606452 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.608853 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:21.613899 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:21.618219 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:21.626247 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.662458 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.683447 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.683804 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:21.687889 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.688165 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:21.696308 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.729559 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.752844 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:21.757746 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:21.781296 12296 consensus_queue.cc:237] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:21.782609 14168 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:21.783169 16390 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:21.785161 16390 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:21.788015 14168 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:21.789175 11887 catalog_manager.cc:5484] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:21.792830 12036 consensus_queue.cc:237] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:21.794145 13906 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:21.794178 12167 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:21.794570 18324 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:21.794749 18311 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:21.796608 18324 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:21.797523 13906 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:21.798854 12167 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:21.801012 11893 catalog_manager.cc:5484] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:21.816238 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet a0106f7a40cb467eb4738890ac4e22d4 with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33028
I20240503 14:00:21.816453 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet 2089eeda1e0f4ff5926584d472e6631d with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:21.817095 12296 consensus_queue.cc:237] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:21.818446 14168 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:21.819119 17566 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:21.819891 12167 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:21.824340 17566 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:21.826395 12167 raft_consensus.cc:858] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:21.826315 17929 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
W20240503 14:00:21.826475 12167 raft_consensus.cc:1565] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:21.826768 12167 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:21.828048 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a0106f7a40cb467eb4738890ac4e22d4 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:21.828893 14168 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:00:21.829855 11891 catalog_manager.cc:5484] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 6 to 7, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:00:21.832775 12167 raft_consensus.cc:1467] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240503 14:00:21.833134 12167 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:00:21.839437 18405 tablet_replica.cc:330] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:21.839640 18405 raft_consensus.cc:2229] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:21.839867 18405 raft_consensus.cc:2258] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:21.841189 18405 ts_tablet_manager.cc:1875] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:00:21.844383 11882 catalog_manager.cc:4804] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): delete failed for tablet a0106f7a40cb467eb4738890ac4e22d4 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 5 but the committed config has opid_index of 7
I20240503 14:00:21.844679 18405 ts_tablet_manager.cc:1888] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:21.844759 18405 log.cc:1196] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2089eeda1e0f4ff5926584d472e6631d
I20240503 14:00:21.847575 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet 2089eeda1e0f4ff5926584d472e6631d (table rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]) successfully deleted
I20240503 14:00:21.933574 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.934983 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.935422 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:21.943646 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:21.945403 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:21.967222 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:21.971139 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:21.987064 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:21.995069 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:21.999305 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.000270 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.011175 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:22.042126 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.043015 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.045656 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.057271 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:22.100139 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:22.101615 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.107196 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.116214 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:22.116361 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.129855 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.130080 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.130610 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:22.132287 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.132493 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.138681 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:22.139423 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:22.139430 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:22.141418 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.157461 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.180260 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.255306 14037 consensus_queue.cc:237] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:22.257481 12428 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer e85b0283a0fd4e3ba81b4170f3cdacb5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:22.257818 12298 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Refusing update from remote peer e85b0283a0fd4e3ba81b4170f3cdacb5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:22.257869 14168 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Refusing update from remote peer e85b0283a0fd4e3ba81b4170f3cdacb5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:22.258612 18648 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:22.259076 17876 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:22.259483 17876 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:22.261873 17876 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:22.263657 12298 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:22.265398 12428 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:22.266690 11890 catalog_manager.cc:5484] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 reported cstate change: config changed from index 5 to 7, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:22.271086 14168 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:22.271617 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c06806a19e4f4592d26ed4acc651cd with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:22.280261 18663 tablet_replica.cc:330] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:22.280616 18663 raft_consensus.cc:2229] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.280846 18663 raft_consensus.cc:2258] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.282986 18663 ts_tablet_manager.cc:1875] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.286170 18663 ts_tablet_manager.cc:1888] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:00:22.286324 18663 log.cc:1196] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f5c06806a19e4f4592d26ed4acc651cd
I20240503 14:00:22.287007 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet f5c06806a19e4f4592d26ed4acc651cd (table rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]) successfully deleted
I20240503 14:00:22.294672 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.305672 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.306476 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:22.306962 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:22.310400 13775 consensus_queue.cc:237] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:22.311909 12298 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:22.312294 17286 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.314297 12167 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:22.314647 17286 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.318207 17286 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:22.321568 11895 catalog_manager.cc:5484] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: config changed from index 4 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:22.321712 12298 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:22.326498 12167 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:22.326773 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.327478 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6ca408c4254678b67a74f49a34881b with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:22.327873 18663 tablet_replica.cc:330] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:22.328011 18663 raft_consensus.cc:2229] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.328123 18663 raft_consensus.cc:2258] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.328735 18663 ts_tablet_manager.cc:1875] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.337487 18663 ts_tablet_manager.cc:1888] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:00:22.337945 18663 log.cc:1196] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5e6ca408c4254678b67a74f49a34881b
I20240503 14:00:22.338578 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet 5e6ca408c4254678b67a74f49a34881b (table rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]) successfully deleted
I20240503 14:00:22.339777 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.342738 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.345412 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:22.374079 12298 consensus_queue.cc:237] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:22.375555 12037 raft_consensus.cc:1261] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.376183 16390 consensus_queue.cc:1035] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.376681 14037 raft_consensus.cc:1261] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.377053 17566 consensus_queue.cc:1035] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.387645 12036 consensus_queue.cc:237] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:22.388089 16390 raft_consensus.cc:2932] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:22.389660 13906 raft_consensus.cc:1261] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.390559 14037 raft_consensus.cc:2932] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:22.391436 18311 consensus_queue.cc:1035] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.391393 11895 catalog_manager.cc:5484] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:22.392441 12167 raft_consensus.cc:1261] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.393179 13775 raft_consensus.cc:1261] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.393671 18324 consensus_queue.cc:1035] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.394465 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e9155230c74e92b6ee441de713a419 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:22.394640 18324 consensus_queue.cc:1035] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.395169 12036 raft_consensus.cc:2932] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:22.401461 18324 raft_consensus.cc:2932] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:22.402442 13775 raft_consensus.cc:2932] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:22.404906 12167 raft_consensus.cc:2932] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:22.407670 11895 catalog_manager.cc:5484] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:22.411669 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2e1517a8ac4041bfb4b90634cb0fa6 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:22.414608 18725 tablet_replica.cc:330] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:22.414757 18725 raft_consensus.cc:2229] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.414912 18725 raft_consensus.cc:2258] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.415984 18725 ts_tablet_manager.cc:1875] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.417093 13906 raft_consensus.cc:2932] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:22.418454 18725 ts_tablet_manager.cc:1888] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:22.418524 18725 log.cc:1196] T d1e9155230c74e92b6ee441de713a419 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d1e9155230c74e92b6ee441de713a419
I20240503 14:00:22.419009 18725 tablet_replica.cc:330] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:22.419091 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet d1e9155230c74e92b6ee441de713a419 (table rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]) successfully deleted
I20240503 14:00:22.419251 18725 raft_consensus.cc:2229] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.419437 18725 raft_consensus.cc:2258] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.420055 18725 ts_tablet_manager.cc:1875] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.422132 18725 ts_tablet_manager.cc:1888] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:22.422821 18725 log.cc:1196] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d2e1517a8ac4041bfb4b90634cb0fa6
I20240503 14:00:22.423390 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet 7d2e1517a8ac4041bfb4b90634cb0fa6 (table rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]) successfully deleted
I20240503 14:00:22.454172 12036 consensus_queue.cc:237] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:22.455572 12428 raft_consensus.cc:1261] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.455601 13906 raft_consensus.cc:1261] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.456045 18324 consensus_queue.cc:1035] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.456256 18722 consensus_queue.cc:1035] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.458544 18324 raft_consensus.cc:2932] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:22.463043 12428 raft_consensus.cc:2932] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:22.465622 11894 catalog_manager.cc:5484] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:22.466943 13906 raft_consensus.cc:2932] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:22.468791 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet dcdc3ae5232a49cdb003d9a2e3cf6d92 with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33028
I20240503 14:00:22.469188 18810 tablet_replica.cc:330] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143: stopping tablet replica
I20240503 14:00:22.469341 18810 raft_consensus.cc:2229] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.469534 18810 raft_consensus.cc:2258] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.470257 18810 ts_tablet_manager.cc:1875] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.482652 18810 ts_tablet_manager.cc:1888] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:22.482847 18810 log.cc:1196] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 8b17d706d2344dbaae13cda9b0733143: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/dcdc3ae5232a49cdb003d9a2e3cf6d92
I20240503 14:00:22.483439 11882 catalog_manager.cc:4830] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): tablet dcdc3ae5232a49cdb003d9a2e3cf6d92 (table rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]) successfully deleted
I20240503 14:00:22.507349 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:22.513695 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.517652 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.521777 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:22.538527 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.543256 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.565577 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:22.566588 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.575990 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.585857 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.594112 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:22.598810 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.628297 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:22.628297 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.639868 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:22.654343 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.667039 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.670614 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:22.687880 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.688849 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:22.705526 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.708278 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.709205 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:22.738126 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.764061 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.766893 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.769759 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.779492 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:22.782397 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:22.789278 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.792918 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.794492 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:22.830201 12298 consensus_queue.cc:237] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:22.831907 12167 raft_consensus.cc:1261] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.832304 18723 consensus_queue.cc:1035] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.835696 14036 raft_consensus.cc:1261] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.835696 14168 raft_consensus.cc:1261] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.836108 18723 consensus_queue.cc:1035] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.836560 16390 consensus_queue.cc:1035] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.862246 16390 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:22.865351 12167 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:22.865418 14036 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:22.865957 14168 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:22.866766 11890 catalog_manager.cc:5484] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:22.871930 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet b031f09b97664d1a94c14f0661181899 with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:22.872511 18983 tablet_replica.cc:330] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:22.872709 18983 raft_consensus.cc:2229] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.873157 18983 raft_consensus.cc:2258] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.873756 18983 ts_tablet_manager.cc:1875] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.882355 18983 ts_tablet_manager.cc:1888] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:22.883435 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.894425 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:22.894222 18983 log.cc:1196] T b031f09b97664d1a94c14f0661181899 P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b031f09b97664d1a94c14f0661181899
I20240503 14:00:22.895031 12428 consensus_queue.cc:237] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:22.897073 14168 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.897913 18515 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.899690 12298 consensus_queue.cc:237] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:22.899890 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.900156 12296 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:22.900527 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:22.901458 14037 raft_consensus.cc:1261] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.901795 12167 raft_consensus.cc:1261] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.902449 18723 consensus_queue.cc:1035] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.902740 18723 consensus_queue.cc:1035] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.907459 13906 raft_consensus.cc:1261] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:22.907848 17566 consensus_queue.cc:1035] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:22.909240 17566 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:22.911082 12167 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:22.911355 14037 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:22.912474 11886 catalog_manager.cc:5484] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:22.913931 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet b031f09b97664d1a94c14f0661181899 (table rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]) successfully deleted
I20240503 14:00:22.917640 18515 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:22.919404 13906 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:22.921447 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet adf2c44328e244779955e712b99f8331 with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:22.921701 18983 tablet_replica.cc:330] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:22.922013 18983 raft_consensus.cc:2229] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.922214 18983 raft_consensus.cc:2258] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.923041 18983 ts_tablet_manager.cc:1875] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.924696 18515 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:22.925596 18983 ts_tablet_manager.cc:1888] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:22.925675 18983 log.cc:1196] T adf2c44328e244779955e712b99f8331 P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/adf2c44328e244779955e712b99f8331
I20240503 14:00:22.926256 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet adf2c44328e244779955e712b99f8331 (table rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]) successfully deleted
I20240503 14:00:22.926983 14168 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:22.927325 12299 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:22.927711 11890 catalog_manager.cc:5484] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 4 to 5, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:22.930790 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet c8933689df2c45b59a9cd16024ddb7d9 with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:33028
I20240503 14:00:22.931010 18810 tablet_replica.cc:330] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143: stopping tablet replica
I20240503 14:00:22.931156 18810 raft_consensus.cc:2229] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:22.931340 18810 raft_consensus.cc:2258] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:22.931959 18810 ts_tablet_manager.cc:1875] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:22.932052 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:22.932833 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:22.934114 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:22.934176 18810 ts_tablet_manager.cc:1888] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:22.934248 18810 log.cc:1196] T c8933689df2c45b59a9cd16024ddb7d9 P 8b17d706d2344dbaae13cda9b0733143: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c8933689df2c45b59a9cd16024ddb7d9
I20240503 14:00:22.935984 11882 catalog_manager.cc:4830] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): tablet c8933689df2c45b59a9cd16024ddb7d9 (table rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]) successfully deleted
I20240503 14:00:22.956750 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:22.999405 12036 consensus_queue.cc:237] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:23.001436 14037 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:23.001629 12167 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:23.001993 13906 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:23.002199 18722 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.002910 18324 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.003223 18324 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.010216 18722 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:23.013228 14037 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:23.013327 13906 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:23.013518 11890 catalog_manager.cc:5484] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:23.014873 12167 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:23.020637 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b4e16598b740cb9d690c2442a3e994 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:23.020990 19093 tablet_replica.cc:330] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:23.021342 19093 raft_consensus.cc:2229] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:23.023706 19093 raft_consensus.cc:2258] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:23.024402 19093 ts_tablet_manager.cc:1875] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:23.027568 19093 ts_tablet_manager.cc:1888] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:23.027643 19093 log.cc:1196] T a4b4e16598b740cb9d690c2442a3e994 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a4b4e16598b740cb9d690c2442a3e994
I20240503 14:00:23.028169 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet a4b4e16598b740cb9d690c2442a3e994 (table rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]) successfully deleted
I20240503 14:00:23.064087 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.064388 12016 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:00:23.064591 13755 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:23.070516 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.072644 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:23.087391 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:23.087165 12299 consensus_queue.cc:237] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:23.088937 13775 raft_consensus.cc:1261] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:23.089527 12428 raft_consensus.cc:1261] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:23.090169 18723 consensus_queue.cc:1035] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:23.090937 18723 consensus_queue.cc:1035] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:23.095815 18723 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:23.098054 13775 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:23.099073 11887 catalog_manager.cc:5484] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 4 to 5, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:23.100365 12428 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:23.104912 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b21b1e5292488aad442060b417f3c1 with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:23.105113 18983 tablet_replica.cc:330] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:23.105242 18983 raft_consensus.cc:2229] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:23.105363 18983 raft_consensus.cc:2258] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:23.105816 18983 ts_tablet_manager.cc:1875] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:23.107901 18983 ts_tablet_manager.cc:1888] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:23.107961 18983 log.cc:1196] T 79b21b1e5292488aad442060b417f3c1 P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/79b21b1e5292488aad442060b417f3c1
I20240503 14:00:23.108493 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet 79b21b1e5292488aad442060b417f3c1 (table rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]) successfully deleted
I20240503 14:00:23.118607 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.119433 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:23.176558 12036 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "355b5456887b4c5694badc9d53b1809a"
dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44052
I20240503 14:00:23.176725 12036 raft_consensus.cc:595] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:23.241672 18311 raft_consensus.cc:979] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: : Instructing follower 66190085d1c04e068d5c0dec9c4bf07d to start an election
I20240503 14:00:23.241796 18311 raft_consensus.cc:1067] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Signalling peer 66190085d1c04e068d5c0dec9c4bf07d to start an election
I20240503 14:00:23.242664 13775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "355b5456887b4c5694badc9d53b1809a"
dest_uuid: "66190085d1c04e068d5c0dec9c4bf07d"
from {username='slave'} at 127.0.99.129:52947
I20240503 14:00:23.242799 13775 raft_consensus.cc:482] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:23.242848 13775 raft_consensus.cc:3037] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:23.244038 13775 raft_consensus.cc:504] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:23.244772 13775 leader_election.cc:288] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [CANDIDATE]: Term 2 election: Requested vote from peers 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), ca5386f84994496fa5376624391728ee (127.0.99.131:36399), b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075)
I20240503 14:00:23.245713 12428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355b5456887b4c5694badc9d53b1809a" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:23.245853 12428 raft_consensus.cc:3037] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:23.247056 12428 raft_consensus.cc:2454] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:23.249958 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355b5456887b4c5694badc9d53b1809a" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:23.250106 12298 raft_consensus.cc:3037] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:23.251353 12298 raft_consensus.cc:2454] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:23.252087 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355b5456887b4c5694badc9d53b1809a" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:23.252224 12036 raft_consensus.cc:3032] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:23.252282 12036 raft_consensus.cc:729] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:23.252475 12036 consensus_queue.cc:260] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:23.252779 12036 raft_consensus.cc:3037] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:23.253934 12036 raft_consensus.cc:2454] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:23.254423 13711 leader_election.cc:302] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 66190085d1c04e068d5c0dec9c4bf07d, 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:23.254642 18704 raft_consensus.cc:2781] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:23.254850 18704 raft_consensus.cc:686] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Becoming Leader. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Running, Role: LEADER
I20240503 14:00:23.255009 18704 consensus_queue.cc:237] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:23.255123 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355b5456887b4c5694badc9d53b1809a" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b67f337e10994234baedbf0b19c13513"
I20240503 14:00:23.255254 13906 raft_consensus.cc:3037] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:23.256301 13906 raft_consensus.cc:2454] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:23.257827 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.260293 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:23.266301 11892 catalog_manager.cc:5484] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: term changed from 1 to 2, leader changed from 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) to 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133). New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:23.282586 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.282773 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:23.284075 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:23.310957 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.319155 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:23.322427 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:23.323048 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.324805 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:23.326237 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.332968 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.333191 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:23.345724 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.364472 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.372344 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.375737 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:23.390893 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.400148 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.405608 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:23.411183 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:23.429967 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.431313 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:23.431996 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:23.435910 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.443446 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.454994 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.469466 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:23.473130 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:23.512550 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:23.531080 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.556502 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.567006 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.569753 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:23.573866 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:23.587725 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.593241 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.608489 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:23.615979 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:23.618403 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:23.654327 12298 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:23.654826 19332 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.664595 12428 consensus_queue.cc:237] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:23.666450 12167 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:23.666939 19008 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.668303 13905 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:23.668768 19008 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.672001 19026 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:23.672394 13905 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:23.676445 11888 catalog_manager.cc:5484] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 4 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:23.679994 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4f3bc64e7145e2b692beab730d1496 with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:23.680559 19459 tablet_replica.cc:330] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:23.680752 19459 raft_consensus.cc:2229] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:23.680727 12167 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:23.681025 19459 raft_consensus.cc:2258] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:23.681905 19459 ts_tablet_manager.cc:1875] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:23.685307 19459 ts_tablet_manager.cc:1888] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:00:23.685420 19459 log.cc:1196] T 6c4f3bc64e7145e2b692beab730d1496 P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6c4f3bc64e7145e2b692beab730d1496
I20240503 14:00:23.685979 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet 6c4f3bc64e7145e2b692beab730d1496 (table rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]) successfully deleted
I20240503 14:00:23.686797 12428 consensus_queue.cc:237] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:23.688405 14037 raft_consensus.cc:1261] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:23.688663 14168 raft_consensus.cc:1261] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:23.688889 12167 raft_consensus.cc:1261] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:23.689080 19008 consensus_queue.cc:1035] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.689293 19471 consensus_queue.cc:1035] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.689607 19026 consensus_queue.cc:1035] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.692224 12428 consensus_queue.cc:237] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:23.693584 14036 raft_consensus.cc:1261] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:23.694006 19026 consensus_queue.cc:1035] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:23.694074 12036 raft_consensus.cc:1261] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:23.694380 19026 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:23.694831 19008 consensus_queue.cc:1035] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:23.697328 12167 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:23.699175 11887 catalog_manager.cc:5484] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:23.700304 14168 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:23.701886 19008 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:23.704675 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet 11a8c270532947ba9b505aace44b56f6 with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:23.704865 19459 tablet_replica.cc:330] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:23.705011 19459 raft_consensus.cc:2229] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:23.705183 19459 raft_consensus.cc:2258] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:23.705813 19459 ts_tablet_manager.cc:1875] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:23.707150 11886 catalog_manager.cc:5484] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:23.707989 19459 ts_tablet_manager.cc:1888] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:23.708052 19459 log.cc:1196] T 11a8c270532947ba9b505aace44b56f6 P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/11a8c270532947ba9b505aace44b56f6
I20240503 14:00:23.709246 14037 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:23.709611 12036 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:23.710170 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet 11a8c270532947ba9b505aace44b56f6 (table rebalance_test_table_2 [id=ef1de2e06d4947918f6cc1607fffc4ee]) successfully deleted
I20240503 14:00:23.710485 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ffc5888a5945d6a8e166420e6f1bd9 with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:23.714186 19502 tablet_replica.cc:330] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:23.714447 19502 raft_consensus.cc:2229] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:23.714795 19502 raft_consensus.cc:2258] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:23.716035 19502 ts_tablet_manager.cc:1875] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:23.716629 14036 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:23.735874 19502 ts_tablet_manager.cc:1888] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:23.735978 19502 log.cc:1196] T 70ffc5888a5945d6a8e166420e6f1bd9 P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/70ffc5888a5945d6a8e166420e6f1bd9
I20240503 14:00:23.736508 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet 70ffc5888a5945d6a8e166420e6f1bd9 (table rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]) successfully deleted
I20240503 14:00:23.765039 12036 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:23.765336 13905 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:23.765566 19332 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.766690 19332 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.772243 12428 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:23.773066 19332 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:23.829636 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:23.838510 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:23.841297 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.842095 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:23.870584 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.870960 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:23.890655 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.898974 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.925774 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.947724 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:23.952816 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.960299 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.963322 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:23.981580 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:23.982223 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.983075 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.995922 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:23.996628 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:23.996694 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:24.019155 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.020730 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:24.026955 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:24.034953 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:24.066417 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:24.091956 12429 consensus_queue.cc:237] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:24.093737 12167 raft_consensus.cc:1261] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:24.094360 14168 raft_consensus.cc:1261] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:24.094705 19471 consensus_queue.cc:1035] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:24.095088 19471 consensus_queue.cc:1035] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:24.098541 19483 raft_consensus.cc:2932] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:24.102775 11887 catalog_manager.cc:5484] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:24.104704 12166 raft_consensus.cc:2932] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:24.107745 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet 211de6f35a844e278e8c1eb1621e7c5f with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:24.107993 19502 tablet_replica.cc:330] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:24.108147 19502 raft_consensus.cc:2229] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:24.108314 19502 raft_consensus.cc:2258] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:24.108980 19502 ts_tablet_manager.cc:1875] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:24.111501 19502 ts_tablet_manager.cc:1888] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:24.111634 19502 log.cc:1196] T 211de6f35a844e278e8c1eb1621e7c5f P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/211de6f35a844e278e8c1eb1621e7c5f
I20240503 14:00:24.112238 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet 211de6f35a844e278e8c1eb1621e7c5f (table rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]) successfully deleted
I20240503 14:00:24.113237 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.113744 14168 raft_consensus.cc:2932] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:24.119215 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.120054 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.122982 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:24.139739 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:24.141616 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:24.143860 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.145973 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:24.175659 12299 consensus_queue.cc:237] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:24.176898 14168 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:24.177443 18723 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:24.179420 18723 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:24.179769 14168 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:24.183043 11894 catalog_manager.cc:5484] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 reported cstate change: config changed from index 7 to 8, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:24.185541 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet a0106f7a40cb467eb4738890ac4e22d4 with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33028
I20240503 14:00:24.186184 12299 consensus_queue.cc:237] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:24.187211 14168 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:24.187325 12166 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:24.187893 18723 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:24.188069 17566 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:24.189514 17566 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:00:24.189811 19800 tablet_replica.cc:330] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: stopping tablet replica
I20240503 14:00:24.189967 19800 raft_consensus.cc:2229] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:24.190083 19800 raft_consensus.cc:2258] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:24.190572 19800 ts_tablet_manager.cc:1875] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:00:24.191535 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): Couldn't send request to peer 8b17d706d2344dbaae13cda9b0733143. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:24.191533 14168 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:00:24.191834 11887 catalog_manager.cc:5484] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 8 to 9, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:00:24.193796 19800 ts_tablet_manager.cc:1888] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 14:00:24.193867 19800 log.cc:1196] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a0106f7a40cb467eb4738890ac4e22d4
I20240503 14:00:24.194068 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a0106f7a40cb467eb4738890ac4e22d4 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:24.194398 11882 catalog_manager.cc:4830] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): tablet a0106f7a40cb467eb4738890ac4e22d4 (table rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]) successfully deleted
I20240503 14:00:24.225497 12299 consensus_queue.cc:237] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:24.227154 12429 raft_consensus.cc:1261] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:24.227162 14168 raft_consensus.cc:1261] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:00:24.227635 18723 consensus_queue.cc:1035] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:24.227852 17566 consensus_queue.cc:1035] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:00:24.238878 18723 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:24.241071 14168 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:24.242771 12429 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:24.242827 11894 catalog_manager.cc:5484] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 4 to 5, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:24.248612 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet e92d8f55591346158b9583feb958a8ca with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:24.248931 19866 tablet_replica.cc:330] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:24.249135 19866 raft_consensus.cc:2229] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:24.249460 19866 raft_consensus.cc:2258] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:24.250186 19866 ts_tablet_manager.cc:1875] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:24.252449 19866 ts_tablet_manager.cc:1888] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:00:24.252511 19866 log.cc:1196] T e92d8f55591346158b9583feb958a8ca P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e92d8f55591346158b9583feb958a8ca
I20240503 14:00:24.252955 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet e92d8f55591346158b9583feb958a8ca (table rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]) successfully deleted
I20240503 14:00:24.347455 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:24.348301 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.355013 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.361937 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.380331 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.383502 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:24.398111 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:24.398373 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.409816 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:24.420238 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:24.421460 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.438303 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:24.449342 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.455710 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.460779 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:24.484601 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:24.487884 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.495760 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:24.503227 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.509759 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:24.511060 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.516497 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.533365 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:24.552794 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:24.565865 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.566087 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:24.581251 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.586789 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.587208 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.604149 20064 ts_tablet_manager.cc:902] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:24.606766 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:24.611778 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:24.614356 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:24.631781 20064 tablet_copy_client.cc:308] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:24.631922 12298 consensus_queue.cc:237] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:24.632460 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet a0106f7a40cb467eb4738890ac4e22d4 from peer 8b17d706d2344dbaae13cda9b0733143 ({username='slave'} at 127.0.99.130:48457)
I20240503 14:00:24.632544 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet a0106f7a40cb467eb4738890ac4e22d4 from peer 8b17d706d2344dbaae13cda9b0733143 at {username='slave'} at 127.0.99.130:48457: session id = 8b17d706d2344dbaae13cda9b0733143-a0106f7a40cb467eb4738890ac4e22d4
I20240503 14:00:24.633040 13775 raft_consensus.cc:1261] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:24.633402 12319 tablet_copy_source_session.cc:205] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:24.633466 17566 consensus_queue.cc:1035] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:24.634524 20064 ts_tablet_manager.cc:1875] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:24.636119 17566 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:24.636647 20064 ts_tablet_manager.cc:1888] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240503 14:00:24.636809 20064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0106f7a40cb467eb4738890ac4e22d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:24.637809 13775 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:24.639288 20064 tablet_copy_client.cc:783] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:24.640065 11894 catalog_manager.cc:5484] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:24.640779 20064 tablet_copy_client.cc:647] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:24.642930 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b21b1e5292488aad442060b417f3c1 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:24.643406 12298 consensus_queue.cc:237] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:24.644419 13775 raft_consensus.cc:1261] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:24.644804 19861 consensus_queue.cc:1035] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:00:24.646031 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79b21b1e5292488aad442060b417f3c1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:24.646138 20064 tablet_copy_client.cc:523] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:24.647544 18723 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:24.649263 20064 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:00:24.649186 13775 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:24.651062 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 79b21b1e5292488aad442060b417f3c1 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:24.651170 11887 catalog_manager.cc:5484] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:24.664950 20082 tablet_replica.cc:330] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:24.665200 20082 raft_consensus.cc:2229] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:24.665899 20082 raft_consensus.cc:2258] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:24.666546 20082 ts_tablet_manager.cc:1875] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:24.669488 20082 ts_tablet_manager.cc:1888] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:24.672793 20082 log.cc:1196] T 79b21b1e5292488aad442060b417f3c1 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/79b21b1e5292488aad442060b417f3c1
I20240503 14:00:24.673617 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet 79b21b1e5292488aad442060b417f3c1 (table rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]) successfully deleted
I20240503 14:00:24.698033 20064 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:24.698452 20064 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap complete.
I20240503 14:00:24.700986 20064 ts_tablet_manager.cc:1367] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.052s user 0.028s sys 0.000s
I20240503 14:00:24.701582 20064 raft_consensus.cc:348] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:24.702216 20064 raft_consensus.cc:729] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:00:24.703979 20064 consensus_queue.cc:260] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:00:24.706990 20064 ts_tablet_manager.cc:1398] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 14:00:24.707543 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 8b17d706d2344dbaae13cda9b0733143-a0106f7a40cb467eb4738890ac4e22d4 received from {username='slave'} at 127.0.99.130:48457
I20240503 14:00:24.707623 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 8b17d706d2344dbaae13cda9b0733143-a0106f7a40cb467eb4738890ac4e22d4 on tablet a0106f7a40cb467eb4738890ac4e22d4 with peer 8b17d706d2344dbaae13cda9b0733143
I20240503 14:00:24.716650 12298 consensus_queue.cc:237] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:24.718210 13775 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:24.718601 19861 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:24.721125 17566 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:24.721529 13775 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:24.722332 13757 consensus_queue.cc:237] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:24.724678 12429 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:24.725557 11886 catalog_manager.cc:5484] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:24.726065 13905 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:24.726496 20146 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:24.726755 20147 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 14:00:24.727802 12036 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:24.729633 19555 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:24.729988 20146 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:24.731839 12036 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:24.732113 13905 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:24.740121 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet 92506cde7b084b14b88d4a915731eeb3 with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:24.740343 19866 tablet_replica.cc:330] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:24.740484 19866 raft_consensus.cc:2229] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:24.740487 11886 catalog_manager.cc:5484] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 reported cstate change: config changed from index 5 to 7, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:24.740658 19866 raft_consensus.cc:2258] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:24.741315 19866 ts_tablet_manager.cc:1875] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:24.744135 19866 ts_tablet_manager.cc:1888] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:24.744201 19866 log.cc:1196] T 92506cde7b084b14b88d4a915731eeb3 P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/92506cde7b084b14b88d4a915731eeb3
I20240503 14:00:24.745836 12428 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:24.745995 12299 consensus_queue.cc:237] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:24.747202 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet 92506cde7b084b14b88d4a915731eeb3 (table rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]) successfully deleted
W20240503 14:00:24.747385 12234 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92506cde7b084b14b88d4a915731eeb3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:24.748226 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet 355b5456887b4c5694badc9d53b1809a with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:24.750145 13757 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:24.750800 20161 tablet_replica.cc:330] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:24.750823 17566 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:24.751087 20161 raft_consensus.cc:2229] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:24.751358 20161 raft_consensus.cc:2258] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:24.753573 17566 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:24.755654 13757 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:24.757033 11892 catalog_manager.cc:5484] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 6 to 7, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:24.758042 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 92506cde7b084b14b88d4a915731eeb3 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:24.763284 20161 ts_tablet_manager.cc:1875] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:24.765985 20161 ts_tablet_manager.cc:1888] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:00:24.766629 20161 log.cc:1196] T 355b5456887b4c5694badc9d53b1809a P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/355b5456887b4c5694badc9d53b1809a
I20240503 14:00:24.768855 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet 355b5456887b4c5694badc9d53b1809a (table rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]) successfully deleted
I20240503 14:00:24.809429 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.813685 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.829391 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:24.833541 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:24.836931 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.847499 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:24.856669 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:24.856905 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:24.859194 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.878865 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:24.883698 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.887893 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.896040 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.910969 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:24.929112 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:24.934960 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.969285 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:24.972254 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:24.983985 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:24.991524 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:24.992286 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:25.009588 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.011024 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:25.030781 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:25.076797 13775 consensus_queue.cc:237] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:25.078815 13905 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:25.079259 20146 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:25.080679 20146 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:25.080973 12428 raft_consensus.cc:1261] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:25.082551 20155 consensus_queue.cc:1035] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:25.083595 12428 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:25.084842 11891 catalog_manager.cc:5484] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: config changed from index 7 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:25.090044 13905 raft_consensus.cc:2932] T 355b5456887b4c5694badc9d53b1809a P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:25.092504 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet 355b5456887b4c5694badc9d53b1809a with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:25.092700 19866 tablet_replica.cc:330] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:25.092828 19866 raft_consensus.cc:2229] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:25.092942 19866 raft_consensus.cc:2258] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:25.093590 19866 ts_tablet_manager.cc:1875] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:25.095818 19866 ts_tablet_manager.cc:1888] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 14:00:25.095894 19866 log.cc:1196] T 355b5456887b4c5694badc9d53b1809a P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/355b5456887b4c5694badc9d53b1809a
I20240503 14:00:25.096371 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet 355b5456887b4c5694badc9d53b1809a (table rebalance_test_table_5 [id=f77268cca2684060a44e936632d05050]) successfully deleted
I20240503 14:00:25.102710 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.105389 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:25.108300 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:25.109871 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:25.120435 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:25.134330 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:25.135942 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.136878 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:25.143510 12166 raft_consensus.cc:1203] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.9] Dedup: 1.9->[]
I20240503 14:00:25.159598 12299 consensus_queue.cc:237] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:25.161041 14168 raft_consensus.cc:1261] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.161041 13775 raft_consensus.cc:1261] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.161379 18723 consensus_queue.cc:1035] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.161463 12166 raft_consensus.cc:1261] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.161864 18723 consensus_queue.cc:1035] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.162156 17566 consensus_queue.cc:1035] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.164649 18723 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:25.165014 12166 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:25.167776 14168 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:25.169675 11893 catalog_manager.cc:5484] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:25.175395 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet df13ec098dc548afac8c67195d25e7e8 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:25.184229 20438 tablet_replica.cc:330] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:25.184461 20438 raft_consensus.cc:2229] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:25.185827 20438 raft_consensus.cc:2258] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:25.187103 20440 ts_tablet_manager.cc:902] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:25.187629 20440 tablet_copy_client.cc:308] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:25.187885 20438 ts_tablet_manager.cc:1875] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:25.188118 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet 79b21b1e5292488aad442060b417f3c1 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:51943)
I20240503 14:00:25.188227 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet 79b21b1e5292488aad442060b417f3c1 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:51943: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-79b21b1e5292488aad442060b417f3c1
I20240503 14:00:25.189273 12319 tablet_copy_source_session.cc:205] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:25.192420 20440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b21b1e5292488aad442060b417f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:25.193255 13775 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:25.195359 20440 tablet_copy_client.cc:783] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:25.195505 20440 tablet_copy_client.cc:647] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:25.198076 20438 ts_tablet_manager.cc:1888] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:25.198211 20438 log.cc:1196] T df13ec098dc548afac8c67195d25e7e8 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/df13ec098dc548afac8c67195d25e7e8
I20240503 14:00:25.198892 20440 tablet_copy_client.cc:523] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:25.201040 20440 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:25.201406 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet df13ec098dc548afac8c67195d25e7e8 (table rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]) successfully deleted
I20240503 14:00:25.250089 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.271297 20440 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:25.272826 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.272431 20440 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:25.273373 20440 ts_tablet_manager.cc:1367] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.072s user 0.028s sys 0.003s
I20240503 14:00:25.282140 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:25.282613 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:25.283329 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:25.289938 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:25.293785 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:25.295497 20440 raft_consensus.cc:348] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:25.295642 20440 raft_consensus.cc:729] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:25.295773 20440 consensus_queue.cc:260] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:25.296079 20440 ts_tablet_manager.cc:1398] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20240503 14:00:25.297310 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-79b21b1e5292488aad442060b417f3c1 received from {username='slave'} at 127.0.99.135:51943
I20240503 14:00:25.297389 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-79b21b1e5292488aad442060b417f3c1 on tablet 79b21b1e5292488aad442060b417f3c1 with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:25.321560 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:25.339123 20591 ts_tablet_manager.cc:902] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:25.339766 20591 tablet_copy_client.cc:308] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:25.341060 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet 92506cde7b084b14b88d4a915731eeb3 from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:41709)
I20240503 14:00:25.341173 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet 92506cde7b084b14b88d4a915731eeb3 from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:41709: session id = b67f337e10994234baedbf0b19c13513-92506cde7b084b14b88d4a915731eeb3
I20240503 14:00:25.342073 12319 tablet_copy_source_session.cc:205] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:25.351929 20591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92506cde7b084b14b88d4a915731eeb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:25.354141 20591 tablet_copy_client.cc:783] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:25.354311 20591 tablet_copy_client.cc:647] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:25.359030 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:25.362237 12148 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:25.370762 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:25.371417 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.375027 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.389061 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:25.395427 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:25.430105 12408 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:00:25.430801 20591 tablet_copy_client.cc:523] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:25.431013 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:25.432317 20591 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:25.433972 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.435108 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.439429 12428 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496"
dest_uuid: "84c63c6f551949f6a1d7c07760824361"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42760
I20240503 14:00:25.439591 12428 raft_consensus.cc:595] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:00:25.477856 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:25.484108 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:25.492523 19471 raft_consensus.cc:979] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: : Instructing follower 8b17d706d2344dbaae13cda9b0733143 to start an election
I20240503 14:00:25.492622 19471 raft_consensus.cc:1067] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 2 LEADER]: Signalling peer 8b17d706d2344dbaae13cda9b0733143 to start an election
I20240503 14:00:25.493292 12166 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496"
dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
from {username='slave'} at 127.0.99.132:45565
I20240503 14:00:25.493490 12166 raft_consensus.cc:482] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:25.493543 12166 raft_consensus.cc:3037] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:00:25.494589 12166 raft_consensus.cc:504] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:25.495011 12166 leader_election.cc:288] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 election: Requested vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075)
I20240503 14:00:25.495607 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496" candidate_uuid: "8b17d706d2344dbaae13cda9b0733143" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "b67f337e10994234baedbf0b19c13513"
I20240503 14:00:25.495711 13905 raft_consensus.cc:3037] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:00:25.496570 13905 raft_consensus.cc:2454] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b17d706d2344dbaae13cda9b0733143 in term 3.
I20240503 14:00:25.497970 12103 leader_election.cc:302] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143, b67f337e10994234baedbf0b19c13513; no voters:
I20240503 14:00:25.498122 19464 raft_consensus.cc:2781] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:00:25.498257 19464 raft_consensus.cc:686] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 3 LEADER]: Becoming Leader. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Running, Role: LEADER
I20240503 14:00:25.498371 19464 consensus_queue.cc:237] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:25.500247 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:25.500968 11891 catalog_manager.cc:5484] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: term changed from 2 to 3, leader changed from 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) to 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:25.504842 20591 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:25.505105 20591 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:25.505280 20591 ts_tablet_manager.cc:1367] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.073s user 0.024s sys 0.001s
I20240503 14:00:25.505759 20591 raft_consensus.cc:348] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:25.505811 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:25.505856 20591 raft_consensus.cc:729] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:25.506003 20591 consensus_queue.cc:260] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:25.506397 20591 ts_tablet_manager.cc:1398] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 14:00:25.506980 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-92506cde7b084b14b88d4a915731eeb3 received from {username='slave'} at 127.0.99.134:41709
I20240503 14:00:25.507054 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session b67f337e10994234baedbf0b19c13513-92506cde7b084b14b88d4a915731eeb3 on tablet 92506cde7b084b14b88d4a915731eeb3 with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:25.508453 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:25.524643 12428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4f3bc64e7145e2b692beab730d1496" candidate_uuid: "8b17d706d2344dbaae13cda9b0733143" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:25.524785 12428 raft_consensus.cc:3032] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:00:25.524854 12428 raft_consensus.cc:729] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Running, Role: LEADER
I20240503 14:00:25.525020 12428 consensus_queue.cc:260] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:25.525256 12428 raft_consensus.cc:3037] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:00:25.526496 12428 raft_consensus.cc:2454] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b17d706d2344dbaae13cda9b0733143 in term 3.
I20240503 14:00:25.605121 12299 consensus_queue.cc:237] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:25.606304 14168 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.606690 19861 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.623611 12299 consensus_queue.cc:237] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:25.624898 14167 raft_consensus.cc:1261] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:25.625285 19861 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.625648 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.632544 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:25.637221 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:25.653188 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.655388 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:25.656100 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:25.673122 19861 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:25.674983 14168 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:25.677107 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.681344 11886 catalog_manager.cc:5484] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:25.686398 12299 consensus_queue.cc:237] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:25.687836 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet a6a89230883347d4a64d96d321247012 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:25.688040 20438 tablet_replica.cc:330] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:25.688189 20438 raft_consensus.cc:2229] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:25.688339 20438 raft_consensus.cc:2258] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:25.688947 20438 ts_tablet_manager.cc:1875] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:00:25.689671 12233 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6a89230883347d4a64d96d321247012. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:25.691046 20438 ts_tablet_manager.cc:1888] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:25.691103 20438 log.cc:1196] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a6a89230883347d4a64d96d321247012
I20240503 14:00:25.691751 14168 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:25.691898 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet a6a89230883347d4a64d96d321247012 (table rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]) successfully deleted
I20240503 14:00:25.692319 19861 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:25.702931 19861 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:25.705250 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:25.705988 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a6a89230883347d4a64d96d321247012 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:25.705998 14168 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:25.707015 11887 catalog_manager.cc:5484] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 6 to 7, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:25.708031 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:25.710359 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:25.728860 14036 raft_consensus.cc:1203] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:00:25.729103 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:25.735675 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:25.774667 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:25.784948 13905 raft_consensus.cc:1203] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:00:25.789269 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:25.793987 19861 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:25.798635 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:25.799693 11887 catalog_manager.cc:5484] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:25.802644 12299 consensus_queue.cc:237] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:25.803483 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet c2e9f6eecbec45988b290775277dbddc with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:25.803773 20438 tablet_replica.cc:330] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:25.803939 20438 raft_consensus.cc:2229] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:25.804090 20438 raft_consensus.cc:2258] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:25.804675 20438 ts_tablet_manager.cc:1875] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:00:25.807478 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2e9f6eecbec45988b290775277dbddc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:25.809240 20438 ts_tablet_manager.cc:1888] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:25.809316 20438 log.cc:1196] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c2e9f6eecbec45988b290775277dbddc
I20240503 14:00:25.809868 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet c2e9f6eecbec45988b290775277dbddc (table rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]) successfully deleted
I20240503 14:00:25.811785 14167 raft_consensus.cc:1261] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:25.812178 17566 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:25.812925 14167 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:25.815766 19861 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:25.818375 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c2e9f6eecbec45988b290775277dbddc with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:25.819445 14167 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:25.820051 11887 catalog_manager.cc:5484] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:25.821000 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:25.936213 13905 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 3 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 14:00:25.936606 20668 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:25.947769 12428 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 3 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 14:00:25.949056 19464 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:25.995883 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.011612 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:26.017694 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:26.018168 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.025317 12299 consensus_queue.cc:237] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:26.029547 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:26.035359 14167 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:00:26.035696 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:26.039402 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:26.040227 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:26.041683 20783 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:26.055016 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:26.059227 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:26.059384 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:26.078399 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:26.080062 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.087877 20783 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.092051 11887 catalog_manager.cc:5484] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 9 to 10, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.097312 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:26.099617 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet a0106f7a40cb467eb4738890ac4e22d4 with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33028
I20240503 14:00:26.099915 12299 consensus_queue.cc:237] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:26.101172 14167 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
W20240503 14:00:26.102054 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0106f7a40cb467eb4738890ac4e22d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:26.102589 14168 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:00:26.103879 19861 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:26.105036 20940 tablet_replica.cc:330] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: stopping tablet replica
I20240503 14:00:26.105172 20940 raft_consensus.cc:2229] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:26.105252 20940 raft_consensus.cc:2258] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.105820 20940 ts_tablet_manager.cc:1875] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:26.108126 20940 ts_tablet_manager.cc:1888] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:00:26.108199 20940 log.cc:1196] T a0106f7a40cb467eb4738890ac4e22d4 P 8b17d706d2344dbaae13cda9b0733143: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a0106f7a40cb467eb4738890ac4e22d4
I20240503 14:00:26.108693 11882 catalog_manager.cc:4830] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): tablet a0106f7a40cb467eb4738890ac4e22d4 (table rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]) successfully deleted
I20240503 14:00:26.112689 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.114096 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:26.125041 19861 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:26.127431 14168 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:26.129771 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a0106f7a40cb467eb4738890ac4e22d4 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:26.136559 11887 catalog_manager.cc:5484] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 reported cstate change: config changed from index 10 to 11, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:26.155416 20962 ts_tablet_manager.cc:902] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:26.156497 20962 tablet_copy_client.cc:308] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:26.156893 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet a6a89230883347d4a64d96d321247012 from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:38411)
I20240503 14:00:26.157003 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet a6a89230883347d4a64d96d321247012 from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:38411: session id = 66190085d1c04e068d5c0dec9c4bf07d-a6a89230883347d4a64d96d321247012
I20240503 14:00:26.157842 12319 tablet_copy_source_session.cc:205] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:26.158614 20962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a89230883347d4a64d96d321247012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:26.160971 20962 tablet_copy_client.cc:783] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:26.161116 20962 tablet_copy_client.cc:647] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:26.162009 20962 tablet_copy_client.cc:523] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:26.163034 20962 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:26.208757 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:26.216801 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.231163 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:26.234375 20962 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:26.234750 20962 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:26.234967 20962 ts_tablet_manager.cc:1367] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.072s user 0.030s sys 0.000s
I20240503 14:00:26.235611 20962 raft_consensus.cc:348] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:26.235756 20962 raft_consensus.cc:729] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:26.240335 20962 consensus_queue.cc:260] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:26.242054 20962 ts_tablet_manager.cc:1398] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:00:26.242532 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-a6a89230883347d4a64d96d321247012 received from {username='slave'} at 127.0.99.133:38411
I20240503 14:00:26.242615 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-a6a89230883347d4a64d96d321247012 on tablet a6a89230883347d4a64d96d321247012 with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:26.245407 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:26.251475 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.268004 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:26.273990 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:26.308746 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:26.317124 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:26.319658 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:26.322310 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:26.336414 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:26.338124 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:26.340587 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.347157 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:26.356940 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.410001 13775 consensus_queue.cc:237] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:26.411455 12166 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:26.411957 20154 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:26.413832 21105 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:26.414242 12166 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:26.417088 11886 catalog_manager.cc:5484] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 6 to 7, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:26.420320 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6ca408c4254678b67a74f49a34881b with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:26.420683 21112 tablet_replica.cc:330] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:26.421684 21112 raft_consensus.cc:2229] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:26.421895 21112 raft_consensus.cc:2258] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.422360 21112 ts_tablet_manager.cc:1875] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:26.426476 21112 ts_tablet_manager.cc:1888] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:00:26.426591 21112 log.cc:1196] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5e6ca408c4254678b67a74f49a34881b
I20240503 14:00:26.427141 21113 ts_tablet_manager.cc:902] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:26.427268 12299 consensus_queue.cc:237] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:26.428050 13775 consensus_queue.cc:237] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:26.428241 14168 raft_consensus.cc:1261] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:26.428718 20783 consensus_queue.cc:1035] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:26.429294 12166 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:26.429700 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet 5e6ca408c4254678b67a74f49a34881b (table rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]) successfully deleted
I20240503 14:00:26.429860 20154 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:26.429908 21113 tablet_copy_client.cc:308] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:26.430405 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet c2e9f6eecbec45988b290775277dbddc from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:51943)
I20240503 14:00:26.430477 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet c2e9f6eecbec45988b290775277dbddc from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:51943: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-c2e9f6eecbec45988b290775277dbddc
W20240503 14:00:26.430465 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e6ca408c4254678b67a74f49a34881b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:26.431279 12319 tablet_copy_source_session.cc:205] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:26.431952 20154 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:26.432189 12166 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:26.432312 19861 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.433565 11880 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5e6ca408c4254678b67a74f49a34881b with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:26.434764 11886 catalog_manager.cc:5484] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: config changed from index 7 to 8, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:26.437753 14168 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.438267 11892 catalog_manager.cc:5484] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 5 to 6, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.438372 21113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e9f6eecbec45988b290775277dbddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:26.442936 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet e92d8f55591346158b9583feb958a8ca with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:26.443157 12299 consensus_queue.cc:237] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:26.443300 21139 tablet_replica.cc:330] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:26.443428 21139 raft_consensus.cc:2229] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 14:00:26.444450 12233 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e92d8f55591346158b9583feb958a8ca. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:26.444823 14168 raft_consensus.cc:1261] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:26.445166 20783 consensus_queue.cc:1035] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:26.445866 21139 raft_consensus.cc:2258] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.446413 21139 ts_tablet_manager.cc:1875] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:26.446544 20783 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:26.447036 21113 tablet_copy_client.cc:783] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:26.447172 21113 tablet_copy_client.cc:647] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:26.447577 14168 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:26.448061 21113 tablet_copy_client.cc:523] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:26.449280 21139 ts_tablet_manager.cc:1888] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:26.449404 21139 log.cc:1196] T e92d8f55591346158b9583feb958a8ca P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/e92d8f55591346158b9583feb958a8ca
I20240503 14:00:26.449954 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e92d8f55591346158b9583feb958a8ca with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:26.449998 11886 catalog_manager.cc:5484] T e92d8f55591346158b9583feb958a8ca P 1b4a61e8a11440e7a9f4b300d2442f75 reported cstate change: config changed from index 6 to 7, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:26.451115 21113 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:26.451308 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet e92d8f55591346158b9583feb958a8ca (table rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]) successfully deleted
I20240503 14:00:26.454483 12166 consensus_queue.cc:237] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:26.455971 13905 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 3 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 14:00:26.456514 19464 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:26.458194 19464 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:26.460202 13905 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:26.462415 11892 catalog_manager.cc:5484] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 6 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 3 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:26.466406 12166 consensus_queue.cc:237] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:26.467639 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4f3bc64e7145e2b692beab730d1496 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:26.467846 21139 tablet_replica.cc:330] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:26.467967 21139 raft_consensus.cc:2229] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:26.468065 21139 raft_consensus.cc:2258] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.468737 13905 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 3 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 14:00:26.468858 21139 ts_tablet_manager.cc:1875] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:26.470731 21157 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:26.472168 21139 ts_tablet_manager.cc:1888] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.7
I20240503 14:00:26.472306 21139 log.cc:1196] T 6c4f3bc64e7145e2b692beab730d1496 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6c4f3bc64e7145e2b692beab730d1496
I20240503 14:00:26.472889 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet 6c4f3bc64e7145e2b692beab730d1496 (table rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]) successfully deleted
I20240503 14:00:26.474257 19464 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.475603 13905 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.476227 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6c4f3bc64e7145e2b692beab730d1496 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:26.479929 11888 catalog_manager.cc:5484] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 8 to 9, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 3 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.487862 21113 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:26.488173 21113 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:26.488348 21113 ts_tablet_manager.cc:1367] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.000s
I20240503 14:00:26.488782 21113 raft_consensus.cc:348] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:26.488904 21113 raft_consensus.cc:729] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:26.489138 21113 consensus_queue.cc:260] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:26.489432 21113 ts_tablet_manager.cc:1398] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:26.489961 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-c2e9f6eecbec45988b290775277dbddc received from {username='slave'} at 127.0.99.135:51943
I20240503 14:00:26.490029 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-c2e9f6eecbec45988b290775277dbddc on tablet c2e9f6eecbec45988b290775277dbddc with peer e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:26.490331 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c4f3bc64e7145e2b692beab730d1496. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:26.551828 21113 ts_tablet_manager.cc:902] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:26.552235 21113 tablet_copy_client.cc:308] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:26.552760 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet a0106f7a40cb467eb4738890ac4e22d4 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 ({username='slave'} at 127.0.99.135:51943)
I20240503 14:00:26.552850 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet a0106f7a40cb467eb4738890ac4e22d4 from peer e85b0283a0fd4e3ba81b4170f3cdacb5 at {username='slave'} at 127.0.99.135:51943: session id = e85b0283a0fd4e3ba81b4170f3cdacb5-a0106f7a40cb467eb4738890ac4e22d4
I20240503 14:00:26.553795 12319 tablet_copy_source_session.cc:205] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:26.554466 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:26.555171 21113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0106f7a40cb467eb4738890ac4e22d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:26.557420 21113 tablet_copy_client.cc:783] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:26.557684 21113 tablet_copy_client.cc:647] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:26.562690 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.569509 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.581643 21113 tablet_copy_client.cc:523] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:26.582809 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:26.583076 21113 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap starting.
I20240503 14:00:26.591854 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:26.593991 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:26.595135 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:26.599581 13775 raft_consensus.cc:1203] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:00:26.613103 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:26.637231 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.649564 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.649937 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:26.659116 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:26.677044 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:26.677917 13755 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:00:26.707840 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:26.711431 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:26.722901 21113 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:26.723205 21113 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5: Bootstrap complete.
I20240503 14:00:26.736181 21113 ts_tablet_manager.cc:1367] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent bootstrapping tablet: real 0.153s user 0.029s sys 0.000s
I20240503 14:00:26.736856 21113 raft_consensus.cc:348] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:26.736979 21113 raft_consensus.cc:729] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b0283a0fd4e3ba81b4170f3cdacb5, State: Initialized, Role: FOLLOWER
I20240503 14:00:26.737576 21113 consensus_queue.cc:260] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:26.739454 21113 ts_tablet_manager.cc:1398] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:26.739929 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-a0106f7a40cb467eb4738890ac4e22d4 received from {username='slave'} at 127.0.99.135:51943
I20240503 14:00:26.740001 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session e85b0283a0fd4e3ba81b4170f3cdacb5-a0106f7a40cb467eb4738890ac4e22d4 on tablet a0106f7a40cb467eb4738890ac4e22d4 with peer e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:26.747174 12299 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3481715d755a4277b0905b0832fad4af"
dest_uuid: "ca5386f84994496fa5376624391728ee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45590
I20240503 14:00:26.747366 12299 raft_consensus.cc:595] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Received request to transfer leadership
W20240503 14:00:26.779537 18648 consensus_queue.cc:1159] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ca5386f84994496fa5376624391728ee" responder_term: 2 status { last_received { term: 2 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 2 index: 7 } }
I20240503 14:00:26.779911 14037 consensus_queue.cc:237] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:26.781126 12428 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer e85b0283a0fd4e3ba81b4170f3cdacb5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:26.781294 14167 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Refusing update from remote peer e85b0283a0fd4e3ba81b4170f3cdacb5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:26.781754 18648 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:26.782003 21445 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:26.783689 18648 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.784078 14167 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.786543 11894 catalog_manager.cc:5484] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 reported cstate change: config changed from index 7 to 8, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.789655 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c06806a19e4f4592d26ed4acc651cd with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:26.789865 21112 tablet_replica.cc:330] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:26.790004 21112 raft_consensus.cc:2229] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:26.790167 21112 raft_consensus.cc:2258] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.790730 21112 ts_tablet_manager.cc:1875] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:26.792335 20797 raft_consensus.cc:979] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee: : Instructing follower 66190085d1c04e068d5c0dec9c4bf07d to start an election
I20240503 14:00:26.792407 20797 raft_consensus.cc:1067] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Signalling peer 66190085d1c04e068d5c0dec9c4bf07d to start an election
I20240503 14:00:26.792990 13775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3481715d755a4277b0905b0832fad4af"
dest_uuid: "66190085d1c04e068d5c0dec9c4bf07d"
from {username='slave'} at 127.0.99.131:46779
I20240503 14:00:26.793097 13775 raft_consensus.cc:482] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:26.793140 13775 raft_consensus.cc:3037] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:26.794138 13775 raft_consensus.cc:504] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:26.794502 13775 leader_election.cc:288] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [CANDIDATE]: Term 2 election: Requested vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:26.794778 12428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3481715d755a4277b0905b0832fad4af" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:26.794883 12428 raft_consensus.cc:3037] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:26.795305 12429 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:26.795867 21112 ts_tablet_manager.cc:1888] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 14:00:26.795897 12428 raft_consensus.cc:2454] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:26.795925 21112 log.cc:1196] T f5c06806a19e4f4592d26ed4acc651cd P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f5c06806a19e4f4592d26ed4acc651cd
I20240503 14:00:26.796252 13708 leader_election.cc:302] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66190085d1c04e068d5c0dec9c4bf07d, 84c63c6f551949f6a1d7c07760824361; no voters:
I20240503 14:00:26.796407 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet f5c06806a19e4f4592d26ed4acc651cd (table rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]) successfully deleted
I20240503 14:00:26.796438 21129 raft_consensus.cc:2781] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:26.796506 21129 raft_consensus.cc:686] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Becoming Leader. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Running, Role: LEADER
I20240503 14:00:26.796634 21129 consensus_queue.cc:237] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:26.797173 12299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3481715d755a4277b0905b0832fad4af" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:26.797307 12299 raft_consensus.cc:3032] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:26.797356 12299 raft_consensus.cc:729] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:26.797497 12299 consensus_queue.cc:260] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:26.797658 12299 raft_consensus.cc:3037] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:26.798629 12299 raft_consensus.cc:2454] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:26.799288 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:26.799728 11893 catalog_manager.cc:5484] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: term changed from 1 to 2, leader changed from ca5386f84994496fa5376624391728ee (127.0.99.131) to 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133). New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:26.799988 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.816960 14037 raft_consensus.cc:1203] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:00:26.819290 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:26.820036 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.823395 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:26.835155 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:26.839920 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:26.840373 21472 ts_tablet_manager.cc:902] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:26.840927 21472 tablet_copy_client.cc:308] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:26.844178 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet e92d8f55591346158b9583feb958a8ca from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:38411)
I20240503 14:00:26.844316 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet e92d8f55591346158b9583feb958a8ca from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:38411: session id = 66190085d1c04e068d5c0dec9c4bf07d-e92d8f55591346158b9583feb958a8ca
I20240503 14:00:26.845207 12319 tablet_copy_source_session.cc:205] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:26.852969 21472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92d8f55591346158b9583feb958a8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:26.856146 21472 tablet_copy_client.cc:783] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:26.856318 21472 tablet_copy_client.cc:647] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:26.857198 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:26.863198 21488 ts_tablet_manager.cc:902] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:26.863808 21488 tablet_copy_client.cc:308] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36263
I20240503 14:00:26.873768 12188 tablet_copy_service.cc:140] P 8b17d706d2344dbaae13cda9b0733143: Received BeginTabletCopySession request for tablet 6c4f3bc64e7145e2b692beab730d1496 from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:51099)
I20240503 14:00:26.873883 12188 tablet_copy_service.cc:161] P 8b17d706d2344dbaae13cda9b0733143: Beginning new tablet copy session on tablet 6c4f3bc64e7145e2b692beab730d1496 from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:51099: session id = 1b4a61e8a11440e7a9f4b300d2442f75-6c4f3bc64e7145e2b692beab730d1496
I20240503 14:00:26.874958 12188 tablet_copy_source_session.cc:205] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:26.875764 21488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4f3bc64e7145e2b692beab730d1496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:26.877889 21488 tablet_copy_client.cc:783] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:26.878046 21488 tablet_copy_client.cc:647] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:26.879500 21488 tablet_copy_client.cc:523] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:26.880615 21488 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:26.893544 21472 tablet_copy_client.cc:523] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:26.895148 21472 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:26.921280 21488 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:26.921660 21488 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:26.921865 21488 ts_tablet_manager.cc:1367] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.041s user 0.026s sys 0.001s
I20240503 14:00:26.922326 21488 raft_consensus.cc:348] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:26.922458 21488 raft_consensus.cc:729] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:26.922626 21488 consensus_queue.cc:260] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:26.923782 21488 ts_tablet_manager.cc:1398] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:00:26.926558 12188 tablet_copy_service.cc:342] P 8b17d706d2344dbaae13cda9b0733143: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-6c4f3bc64e7145e2b692beab730d1496 received from {username='slave'} at 127.0.99.136:51099
I20240503 14:00:26.926666 12188 tablet_copy_service.cc:433] P 8b17d706d2344dbaae13cda9b0733143: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-6c4f3bc64e7145e2b692beab730d1496 on tablet 6c4f3bc64e7145e2b692beab730d1496 with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:26.942131 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.957211 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:26.964104 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:26.967453 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:26.985296 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:26.990788 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:26.991446 21551 ts_tablet_manager.cc:902] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005)
I20240503 14:00:26.992163 21551 tablet_copy_client.cc:308] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37005
I20240503 14:00:27.004717 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:27.005029 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:27.026232 13795 tablet_copy_service.cc:140] P 66190085d1c04e068d5c0dec9c4bf07d: Received BeginTabletCopySession request for tablet 5e6ca408c4254678b67a74f49a34881b from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:39173)
I20240503 14:00:27.026364 13795 tablet_copy_service.cc:161] P 66190085d1c04e068d5c0dec9c4bf07d: Beginning new tablet copy session on tablet 5e6ca408c4254678b67a74f49a34881b from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:39173: session id = b67f337e10994234baedbf0b19c13513-5e6ca408c4254678b67a74f49a34881b
I20240503 14:00:27.027738 13795 tablet_copy_source_session.cc:205] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:27.029400 21551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6ca408c4254678b67a74f49a34881b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:27.032083 21551 tablet_copy_client.cc:783] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:27.032297 21551 tablet_copy_client.cc:647] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:27.037115 21551 tablet_copy_client.cc:523] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:27.038456 21551 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:27.045313 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.045751 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:27.050995 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.055464 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.062223 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.069484 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.076186 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.078333 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:27.086751 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:27.096599 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:27.097204 14037 raft_consensus.cc:1203] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 14:00:27.101670 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:27.118690 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:27.122902 21551 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:27.123239 21551 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:27.123469 21551 ts_tablet_manager.cc:1367] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.085s user 0.026s sys 0.000s
I20240503 14:00:27.123962 21551 raft_consensus.cc:348] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:27.124145 21551 raft_consensus.cc:729] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:27.124332 21551 consensus_queue.cc:260] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:27.124941 21551 ts_tablet_manager.cc:1398] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:27.125538 13795 tablet_copy_service.cc:342] P 66190085d1c04e068d5c0dec9c4bf07d: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-5e6ca408c4254678b67a74f49a34881b received from {username='slave'} at 127.0.99.134:39173
I20240503 14:00:27.125633 13795 tablet_copy_service.cc:433] P 66190085d1c04e068d5c0dec9c4bf07d: ending tablet copy session b67f337e10994234baedbf0b19c13513-5e6ca408c4254678b67a74f49a34881b on tablet 5e6ca408c4254678b67a74f49a34881b with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:27.128134 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.128577 21472 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:27.128930 21472 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:27.129173 21472 ts_tablet_manager.cc:1367] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.234s user 0.030s sys 0.000s
I20240503 14:00:27.129833 21472 raft_consensus.cc:348] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:27.130002 21472 raft_consensus.cc:729] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:27.130189 21472 consensus_queue.cc:260] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:27.130599 21472 ts_tablet_manager.cc:1398] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:27.130980 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-e92d8f55591346158b9583feb958a8ca received from {username='slave'} at 127.0.99.133:38411
I20240503 14:00:27.131043 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-e92d8f55591346158b9583feb958a8ca on tablet e92d8f55591346158b9583feb958a8ca with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:27.139657 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.141034 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:27.142419 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:27.150504 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.153661 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:27.179934 12429 consensus_queue.cc:237] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:27.180858 14037 raft_consensus.cc:1261] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:27.181227 20667 consensus_queue.cc:1035] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:27.187794 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.190960 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.193311 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:27.195597 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:27.204064 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.205718 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:27.210841 20667 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:27.214831 11894 catalog_manager.cc:5484] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:27.216732 14037 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:27.222075 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ffc5888a5945d6a8e166420e6f1bd9 with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:27.222385 21640 tablet_replica.cc:330] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:27.222580 21640 raft_consensus.cc:2229] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:27.222800 21640 raft_consensus.cc:2258] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:27.222741 12429 consensus_queue.cc:237] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:27.223647 21640 ts_tablet_manager.cc:1875] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:27.223819 12429 raft_consensus.cc:1261] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240503 14:00:27.224246 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70ffc5888a5945d6a8e166420e6f1bd9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:27.224527 21129 consensus_queue.cc:1035] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:27.224643 14037 raft_consensus.cc:1261] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:27.225100 20667 consensus_queue.cc:1035] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:27.226898 21640 ts_tablet_manager.cc:1888] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:27.227003 21640 log.cc:1196] T 70ffc5888a5945d6a8e166420e6f1bd9 P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/70ffc5888a5945d6a8e166420e6f1bd9
I20240503 14:00:27.227564 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet 70ffc5888a5945d6a8e166420e6f1bd9 (table rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]) successfully deleted
I20240503 14:00:27.229647 12299 raft_consensus.cc:1261] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:27.230150 21129 consensus_queue.cc:1035] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:27.242424 20667 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:27.245134 14037 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:27.246222 11879 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 70ffc5888a5945d6a8e166420e6f1bd9 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:27.246805 11891 catalog_manager.cc:5484] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 6 to 7, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:27.278036 12429 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "211de6f35a844e278e8c1eb1621e7c5f"
dest_uuid: "84c63c6f551949f6a1d7c07760824361"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43044
I20240503 14:00:27.278156 12429 raft_consensus.cc:595] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:27.279026 12166 consensus_queue.cc:237] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:27.280251 13905 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:27.280652 21157 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:27.291924 21157 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:27.295399 11891 catalog_manager.cc:5484] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:27.295468 13905 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:27.298516 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet c42fee9fadbb4601a7fe32eb6a04e767 with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:27.298710 21640 tablet_replica.cc:330] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:27.298820 21640 raft_consensus.cc:2229] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:27.298945 21640 raft_consensus.cc:2258] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:27.299888 12166 consensus_queue.cc:237] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:27.302736 21640 ts_tablet_manager.cc:1875] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:27.306917 21640 ts_tablet_manager.cc:1888] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:00:27.306994 21640 log.cc:1196] T c42fee9fadbb4601a7fe32eb6a04e767 P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c42fee9fadbb4601a7fe32eb6a04e767
W20240503 14:00:27.310691 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c42fee9fadbb4601a7fe32eb6a04e767. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:27.310958 13905 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:27.311404 21157 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:27.313513 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet c42fee9fadbb4601a7fe32eb6a04e767 (table rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]) successfully deleted
I20240503 14:00:27.317561 21157 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:27.318153 12299 consensus_queue.cc:237] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:27.319329 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c42fee9fadbb4601a7fe32eb6a04e767 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:27.319916 14037 raft_consensus.cc:1261] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:27.320596 11892 catalog_manager.cc:5484] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 6 to 7, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:27.321347 17566 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:27.322248 13905 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
W20240503 14:00:27.323177 12232 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): Couldn't send request to peer 84c63c6f551949f6a1d7c07760824361. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:27.323907 14167 raft_consensus.cc:1261] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:27.324286 13775 raft_consensus.cc:1203] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:00:27.326398 20783 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:27.330518 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.337605 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:27.338045 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.340903 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:27.356645 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.369625 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:27.371841 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.391893 20783 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:27.394611 14167 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:27.396152 14036 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:27.396641 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:27.397893 11894 catalog_manager.cc:5484] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:27.430286 12429 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "211de6f35a844e278e8c1eb1621e7c5f"
dest_uuid: "84c63c6f551949f6a1d7c07760824361"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43090
I20240503 14:00:27.430457 12429 raft_consensus.cc:595] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:27.452756 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.455163 12298 consensus_queue.cc:237] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:27.456744 13905 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:27.457921 13775 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:27.458444 21738 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:00:27.458906 12232 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee -> Peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): Couldn't send request to peer 84c63c6f551949f6a1d7c07760824361. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:27.459698 21738 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:27.462446 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:27.463013 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.470351 21738 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:27.472790 13905 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:27.474515 13775 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:27.475972 11891 catalog_manager.cc:5484] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:27.477728 14167 raft_consensus.cc:1203] T 6c4f3bc64e7145e2b692beab730d1496 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9] Dedup: 3.9->[]
I20240503 14:00:27.481384 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.484150 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.506613 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:27.507370 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:27.526607 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:27.533417 13905 raft_consensus.cc:1203] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:00:27.558395 12298 consensus_queue.cc:237] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:27.559808 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.559939 14036 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:00:27.560266 14167 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:00:27.560597 20783 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 14:00:27.560823 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0106f7a40cb467eb4738890ac4e22d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:27.560979 20783 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:00:27.565452 14148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.566004 12408 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:00:27.573552 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:27.573791 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.586508 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.597751 21771 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:27.598310 14036 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:27.600113 14167 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:27.601612 11886 catalog_manager.cc:5484] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 11 to 12, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:27.614223 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:27.615696 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:27.653657 21652 raft_consensus.cc:979] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361: : Instructing follower 1b4a61e8a11440e7a9f4b300d2442f75 to start an election
I20240503 14:00:27.653774 21652 raft_consensus.cc:1067] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Signalling peer 1b4a61e8a11440e7a9f4b300d2442f75 to start an election
I20240503 14:00:27.654284 14167 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "211de6f35a844e278e8c1eb1621e7c5f"
dest_uuid: "1b4a61e8a11440e7a9f4b300d2442f75"
from {username='slave'} at 127.0.99.132:42779
I20240503 14:00:27.654505 14167 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:27.654592 14167 raft_consensus.cc:3037] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:27.655771 14167 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:27.656839 14167 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [CANDIDATE]: Term 2 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:27.657318 12166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211de6f35a844e278e8c1eb1621e7c5f" candidate_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:27.657438 12166 raft_consensus.cc:3037] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:27.658278 12166 raft_consensus.cc:2454] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4a61e8a11440e7a9f4b300d2442f75 in term 2.
I20240503 14:00:27.658849 14104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4a61e8a11440e7a9f4b300d2442f75, 8b17d706d2344dbaae13cda9b0733143; no voters:
I20240503 14:00:27.659075 20777 raft_consensus.cc:2781] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:27.659209 20777 raft_consensus.cc:686] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 LEADER]: Becoming Leader. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Running, Role: LEADER
I20240503 14:00:27.659595 20777 consensus_queue.cc:237] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:27.661680 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211de6f35a844e278e8c1eb1621e7c5f" candidate_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:27.661782 12429 raft_consensus.cc:3032] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:27.661813 12429 raft_consensus.cc:729] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Running, Role: LEADER
I20240503 14:00:27.661912 12429 consensus_queue.cc:260] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:27.662037 12429 raft_consensus.cc:3037] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:27.662321 11886 catalog_manager.cc:5484] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 reported cstate change: term changed from 1 to 2, leader changed from 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) to 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:27.662910 12429 raft_consensus.cc:2454] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4a61e8a11440e7a9f4b300d2442f75 in term 2.
I20240503 14:00:27.678195 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:27.679469 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:27.685194 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.686895 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.690778 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:27.692515 12408 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:27.699605 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.708137 12166 consensus_queue.cc:237] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:27.710619 14036 raft_consensus.cc:1261] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:27.710956 12036 raft_consensus.cc:1261] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:27.711089 21712 consensus_queue.cc:1035] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:27.711341 21713 consensus_queue.cc:1035] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:27.713716 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 14:00:27.714041 12102 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4f9cb87c70e43dbb0d850c275ebb8a5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:27.743855 21713 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:27.746968 12036 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:27.747853 11886 catalog_manager.cc:5484] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 5 to 6, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:27.751516 14037 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:27.792479 12298 consensus_queue.cc:237] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:27.794492 12036 raft_consensus.cc:1261] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:27.794548 14037 raft_consensus.cc:1261] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 14:00:27.794725 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1e9155230c74e92b6ee441de713a419. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:27.795030 21771 consensus_queue.cc:1035] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:27.795372 20783 consensus_queue.cc:1035] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:27.812534 22069 ts_tablet_manager.cc:902] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:27.813618 22069 tablet_copy_client.cc:308] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:27.814234 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet c2e9f6eecbec45988b290775277dbddc from peer 84c63c6f551949f6a1d7c07760824361 ({username='slave'} at 127.0.99.132:33737)
I20240503 14:00:27.814319 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet c2e9f6eecbec45988b290775277dbddc from peer 84c63c6f551949f6a1d7c07760824361 at {username='slave'} at 127.0.99.132:33737: session id = 84c63c6f551949f6a1d7c07760824361-c2e9f6eecbec45988b290775277dbddc
I20240503 14:00:27.815277 12319 tablet_copy_source_session.cc:205] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:27.816995 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:27.817083 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.817479 12408 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:27.818611 22069 ts_tablet_manager.cc:1875] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:27.820612 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:27.827395 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:27.827701 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:27.840700 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:27.844791 22082 ts_tablet_manager.cc:902] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:27.845237 22082 tablet_copy_client.cc:308] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33927
I20240503 14:00:27.846609 22069 ts_tablet_manager.cc:1888] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 14:00:27.846773 22069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e9f6eecbec45988b290775277dbddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:27.847560 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:27.848258 22069 tablet_copy_client.cc:783] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:27.848424 22069 tablet_copy_client.cc:647] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:27.849993 22069 tablet_copy_client.cc:523] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:27.851485 22069 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:27.860225 22058 ts_tablet_manager.cc:902] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:27.860726 22058 tablet_copy_client.cc:308] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36263
I20240503 14:00:27.863972 12188 tablet_copy_service.cc:140] P 8b17d706d2344dbaae13cda9b0733143: Received BeginTabletCopySession request for tablet c42fee9fadbb4601a7fe32eb6a04e767 from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:51099)
I20240503 14:00:27.864086 12188 tablet_copy_service.cc:161] P 8b17d706d2344dbaae13cda9b0733143: Beginning new tablet copy session on tablet c42fee9fadbb4601a7fe32eb6a04e767 from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:51099: session id = 1b4a61e8a11440e7a9f4b300d2442f75-c42fee9fadbb4601a7fe32eb6a04e767
I20240503 14:00:27.864980 12188 tablet_copy_source_session.cc:205] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:27.865811 20783 raft_consensus.cc:2932] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:27.867168 22058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42fee9fadbb4601a7fe32eb6a04e767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:27.868957 12036 raft_consensus.cc:2932] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:27.869347 11887 catalog_manager.cc:5484] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 5 to 6, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:27.871332 22058 tablet_copy_client.cc:783] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:27.871474 22058 tablet_copy_client.cc:647] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:27.871640 14037 raft_consensus.cc:2932] T d1e9155230c74e92b6ee441de713a419 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:27.879794 12450 tablet_copy_service.cc:140] P 84c63c6f551949f6a1d7c07760824361: Received BeginTabletCopySession request for tablet 70ffc5888a5945d6a8e166420e6f1bd9 from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:51623)
I20240503 14:00:27.879878 12450 tablet_copy_service.cc:161] P 84c63c6f551949f6a1d7c07760824361: Beginning new tablet copy session on tablet 70ffc5888a5945d6a8e166420e6f1bd9 from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:51623: session id = b67f337e10994234baedbf0b19c13513-70ffc5888a5945d6a8e166420e6f1bd9
I20240503 14:00:27.880712 12450 tablet_copy_source_session.cc:205] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:27.881495 22082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ffc5888a5945d6a8e166420e6f1bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:27.883653 22082 tablet_copy_client.cc:783] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:27.883896 22082 tablet_copy_client.cc:647] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:27.886559 22069 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:27.886581 22082 tablet_copy_client.cc:523] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:27.887332 22069 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Bootstrap complete.
I20240503 14:00:27.887557 22069 ts_tablet_manager.cc:1367] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.036s user 0.025s sys 0.000s
I20240503 14:00:27.887961 22069 raft_consensus.cc:348] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:27.888062 22069 raft_consensus.cc:729] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:27.888178 22069 consensus_queue.cc:260] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:27.888396 22069 ts_tablet_manager.cc:1398] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:27.888571 22058 tablet_copy_client.cc:523] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:27.889007 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 84c63c6f551949f6a1d7c07760824361-c2e9f6eecbec45988b290775277dbddc received from {username='slave'} at 127.0.99.132:33737
I20240503 14:00:27.889070 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 84c63c6f551949f6a1d7c07760824361-c2e9f6eecbec45988b290775277dbddc on tablet c2e9f6eecbec45988b290775277dbddc with peer 84c63c6f551949f6a1d7c07760824361
I20240503 14:00:27.889364 22082 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:27.890452 22058 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:27.892726 22069 ts_tablet_manager.cc:902] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:27.893111 22069 tablet_copy_client.cc:308] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:27.893488 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet 92506cde7b084b14b88d4a915731eeb3 from peer 84c63c6f551949f6a1d7c07760824361 ({username='slave'} at 127.0.99.132:33737)
I20240503 14:00:27.893582 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet 92506cde7b084b14b88d4a915731eeb3 from peer 84c63c6f551949f6a1d7c07760824361 at {username='slave'} at 127.0.99.132:33737: session id = 84c63c6f551949f6a1d7c07760824361-92506cde7b084b14b88d4a915731eeb3
I20240503 14:00:27.894472 12319 tablet_copy_source_session.cc:205] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:27.895175 22069 ts_tablet_manager.cc:1875] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:27.897557 22069 ts_tablet_manager.cc:1888] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 14:00:27.897766 22069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92506cde7b084b14b88d4a915731eeb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:27.898782 22069 tablet_copy_client.cc:783] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:27.898932 22069 tablet_copy_client.cc:647] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:27.905087 22069 tablet_copy_client.cc:523] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:27.906102 22069 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:27.925050 22058 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:27.925359 22058 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:27.925555 22058 ts_tablet_manager.cc:1367] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.003s
I20240503 14:00:27.926370 22058 raft_consensus.cc:348] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:27.926538 22058 raft_consensus.cc:729] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:27.926707 22058 consensus_queue.cc:260] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:27.927433 22058 ts_tablet_manager.cc:1398] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:27.933498 12188 tablet_copy_service.cc:342] P 8b17d706d2344dbaae13cda9b0733143: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-c42fee9fadbb4601a7fe32eb6a04e767 received from {username='slave'} at 127.0.99.136:51099
I20240503 14:00:27.933580 12188 tablet_copy_service.cc:433] P 8b17d706d2344dbaae13cda9b0733143: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-c42fee9fadbb4601a7fe32eb6a04e767 on tablet c42fee9fadbb4601a7fe32eb6a04e767 with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:27.950079 22082 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:27.950464 22082 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:27.950611 22082 ts_tablet_manager.cc:1367] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.061s user 0.027s sys 0.000s
I20240503 14:00:27.950979 22082 raft_consensus.cc:348] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:27.951093 22082 raft_consensus.cc:729] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:27.951223 22082 consensus_queue.cc:260] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:27.951571 22082 ts_tablet_manager.cc:1398] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:27.951997 12450 tablet_copy_service.cc:342] P 84c63c6f551949f6a1d7c07760824361: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-70ffc5888a5945d6a8e166420e6f1bd9 received from {username='slave'} at 127.0.99.134:51623
I20240503 14:00:27.952085 12450 tablet_copy_service.cc:433] P 84c63c6f551949f6a1d7c07760824361: ending tablet copy session b67f337e10994234baedbf0b19c13513-70ffc5888a5945d6a8e166420e6f1bd9 on tablet 70ffc5888a5945d6a8e166420e6f1bd9 with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:27.957984 13775 consensus_queue.cc:237] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:27.959307 12429 raft_consensus.cc:1261] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:27.959733 21129 consensus_queue.cc:1035] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:27.962868 21129 raft_consensus.cc:2932] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:27.965015 12429 raft_consensus.cc:2932] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:27.971149 11890 catalog_manager.cc:5484] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: config changed from index 5 to 7, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:00:27.974015 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet 3481715d755a4277b0905b0832fad4af with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:27.975076 13775 consensus_queue.cc:237] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
W20240503 14:00:27.976786 13711 consensus_peers.cc:485] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d -> Peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): Couldn't send request to peer 623e8dcb58544e558207d8e652dfed5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3481715d755a4277b0905b0832fad4af. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:27.977037 12429 raft_consensus.cc:1261] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:27.977689 21129 consensus_queue.cc:1035] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:27.979223 21129 raft_consensus.cc:2932] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:27.980986 12429 raft_consensus.cc:2932] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:27.983564 11888 catalog_manager.cc:5484] T 3481715d755a4277b0905b0832fad4af P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 7 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:27.984418 11880 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 3481715d755a4277b0905b0832fad4af with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:27.992535 22191 tablet_replica.cc:330] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:27.993345 22191 raft_consensus.cc:2229] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:27.993518 22191 raft_consensus.cc:2258] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:27.994203 22191 ts_tablet_manager.cc:1875] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:27.996240 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.001941 22191 ts_tablet_manager.cc:1888] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:00:28.004099 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:28.004292 22191 log.cc:1196] T 3481715d755a4277b0905b0832fad4af P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3481715d755a4277b0905b0832fad4af
I20240503 14:00:28.004853 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet 3481715d755a4277b0905b0832fad4af (table rebalance_test_table_8 [id=22cca43e02d54fad8bd2569ae6b8bd0f]) successfully deleted
I20240503 14:00:28.011132 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.017921 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.021322 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:28.021685 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.027683 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.029955 22069 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:28.030262 22069 tablet_bootstrap.cc:492] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Bootstrap complete.
I20240503 14:00:28.030591 22069 ts_tablet_manager.cc:1367] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.125s user 0.021s sys 0.004s
I20240503 14:00:28.031075 22069 raft_consensus.cc:348] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:28.031389 22069 raft_consensus.cc:729] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:28.038223 22069 consensus_queue.cc:260] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:28.039165 22069 ts_tablet_manager.cc:1398] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 14:00:28.039626 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 84c63c6f551949f6a1d7c07760824361-92506cde7b084b14b88d4a915731eeb3 received from {username='slave'} at 127.0.99.132:33737
I20240503 14:00:28.039707 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 84c63c6f551949f6a1d7c07760824361-92506cde7b084b14b88d4a915731eeb3 on tablet 92506cde7b084b14b88d4a915731eeb3 with peer 84c63c6f551949f6a1d7c07760824361
I20240503 14:00:28.041287 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:28.047672 12408 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:28.047776 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.066381 12429 raft_consensus.cc:1261] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer 1b4a61e8a11440e7a9f4b300d2442f75: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:28.066900 20777 consensus_queue.cc:1035] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:28.070219 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:28.076994 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.079675 12408 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:28.086548 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:28.095541 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.106493 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:28.109304 12166 raft_consensus.cc:1261] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer 1b4a61e8a11440e7a9f4b300d2442f75: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:28.109778 21959 consensus_queue.cc:1035] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:28.153484 22082 ts_tablet_manager.cc:902] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:28.153931 22082 tablet_copy_client.cc:308] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:28.154429 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet a0106f7a40cb467eb4738890ac4e22d4 from peer b67f337e10994234baedbf0b19c13513 ({username='slave'} at 127.0.99.134:41709)
I20240503 14:00:28.154537 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet a0106f7a40cb467eb4738890ac4e22d4 from peer b67f337e10994234baedbf0b19c13513 at {username='slave'} at 127.0.99.134:41709: session id = b67f337e10994234baedbf0b19c13513-a0106f7a40cb467eb4738890ac4e22d4
I20240503 14:00:28.155558 12319 tablet_copy_source_session.cc:205] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:28.156447 22082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0106f7a40cb467eb4738890ac4e22d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:28.158421 22082 tablet_copy_client.cc:783] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:28.158560 22082 tablet_copy_client.cc:647] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:28.162981 12429 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c8933689df2c45b59a9cd16024ddb7d9"
dest_uuid: "84c63c6f551949f6a1d7c07760824361"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43180
I20240503 14:00:28.163100 12429 raft_consensus.cc:595] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:28.168037 22082 tablet_copy_client.cc:523] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:28.169586 22082 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513: Bootstrap starting.
I20240503 14:00:28.181743 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.187963 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:28.192199 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.192837 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.192878 14167 raft_consensus.cc:1203] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:00:28.202930 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:28.207872 12429 raft_consensus.cc:1203] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:28.212277 12408 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:28.214149 13905 raft_consensus.cc:1203] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:00:28.222371 22082 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:28.222704 22082 tablet_bootstrap.cc:492] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513: Bootstrap complete.
I20240503 14:00:28.222906 22082 ts_tablet_manager.cc:1367] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513: Time spent bootstrapping tablet: real 0.053s user 0.026s sys 0.004s
I20240503 14:00:28.223387 22082 raft_consensus.cc:348] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:28.223549 22082 raft_consensus.cc:729] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67f337e10994234baedbf0b19c13513, State: Initialized, Role: FOLLOWER
I20240503 14:00:28.223714 22082 consensus_queue.cc:260] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:28.224066 22082 ts_tablet_manager.cc:1398] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:28.225230 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:28.230643 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session b67f337e10994234baedbf0b19c13513-a0106f7a40cb467eb4738890ac4e22d4 received from {username='slave'} at 127.0.99.134:41709
I20240503 14:00:28.230731 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session b67f337e10994234baedbf0b19c13513-a0106f7a40cb467eb4738890ac4e22d4 on tablet a0106f7a40cb467eb4738890ac4e22d4 with peer b67f337e10994234baedbf0b19c13513
I20240503 14:00:28.247421 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.255757 22319 ts_tablet_manager.cc:902] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:28.263595 22319 tablet_copy_client.cc:308] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36263
I20240503 14:00:28.264091 12188 tablet_copy_service.cc:140] P 8b17d706d2344dbaae13cda9b0733143: Received BeginTabletCopySession request for tablet f4f9cb87c70e43dbb0d850c275ebb8a5 from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:52409)
I20240503 14:00:28.264214 12188 tablet_copy_service.cc:161] P 8b17d706d2344dbaae13cda9b0733143: Beginning new tablet copy session on tablet f4f9cb87c70e43dbb0d850c275ebb8a5 from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:52409: session id = 66190085d1c04e068d5c0dec9c4bf07d-f4f9cb87c70e43dbb0d850c275ebb8a5
I20240503 14:00:28.265174 12188 tablet_copy_source_session.cc:205] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:28.266103 22319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f9cb87c70e43dbb0d850c275ebb8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:28.269809 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.270471 22319 tablet_copy_client.cc:783] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:28.270498 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:28.270632 22319 tablet_copy_client.cc:647] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:28.287210 22319 tablet_copy_client.cc:523] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:28.295863 22319 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:00:28.302584 12277 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:00:28.305248 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:28.305248 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.313051 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.318572 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.326270 12408 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:28.327443 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:28.327850 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.330636 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.339556 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.350399 12408 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:28.354540 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:28.371054 21633 raft_consensus.cc:979] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361: : Instructing follower ca5386f84994496fa5376624391728ee to start an election
I20240503 14:00:28.371155 21633 raft_consensus.cc:1067] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Signalling peer ca5386f84994496fa5376624391728ee to start an election
I20240503 14:00:28.371635 12298 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8933689df2c45b59a9cd16024ddb7d9"
dest_uuid: "ca5386f84994496fa5376624391728ee"
from {username='slave'} at 127.0.99.132:33737
I20240503 14:00:28.371821 12298 raft_consensus.cc:482] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:28.372071 12298 raft_consensus.cc:3037] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:28.373287 12298 raft_consensus.cc:504] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:28.374131 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8933689df2c45b59a9cd16024ddb7d9" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:28.374238 12429 raft_consensus.cc:3032] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:28.374272 12429 raft_consensus.cc:729] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Running, Role: LEADER
I20240503 14:00:28.374387 12429 consensus_queue.cc:260] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:28.374725 12429 raft_consensus.cc:3037] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:28.375797 12429 raft_consensus.cc:2454] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 2.
I20240503 14:00:28.376435 12232 leader_election.cc:302] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:28.376641 21772 raft_consensus.cc:2781] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:28.376706 21772 raft_consensus.cc:686] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 2 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:28.376830 21772 consensus_queue.cc:237] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:28.381704 11890 catalog_manager.cc:5484] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 1 to 2, leader changed from 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 2 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:28.382544 12298 leader_election.cc:288] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 2 election: Requested vote from peers 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763)
I20240503 14:00:28.382994 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8933689df2c45b59a9cd16024ddb7d9" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1b4a61e8a11440e7a9f4b300d2442f75"
I20240503 14:00:28.383102 14167 raft_consensus.cc:3037] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:28.384130 14167 raft_consensus.cc:2454] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 2.
I20240503 14:00:28.386080 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:28.400835 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.416254 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.416678 22058 ts_tablet_manager.cc:902] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:28.417054 22058 tablet_copy_client.cc:308] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:28.417398 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet d1e9155230c74e92b6ee441de713a419 from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:41157)
I20240503 14:00:28.417493 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet d1e9155230c74e92b6ee441de713a419 from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:41157: session id = 1b4a61e8a11440e7a9f4b300d2442f75-d1e9155230c74e92b6ee441de713a419
I20240503 14:00:28.418306 12319 tablet_copy_source_session.cc:205] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:28.419129 22058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e9155230c74e92b6ee441de713a419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:28.421141 22058 tablet_copy_client.cc:783] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:28.421284 22058 tablet_copy_client.cc:647] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:28.427520 22058 tablet_copy_client.cc:523] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:28.428620 22058 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
I20240503 14:00:28.436142 22397 ts_tablet_manager.cc:902] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f: Initiating tablet copy from peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005)
I20240503 14:00:28.436829 22397 tablet_copy_client.cc:308] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37005
I20240503 14:00:28.437288 13795 tablet_copy_service.cc:140] P 66190085d1c04e068d5c0dec9c4bf07d: Received BeginTabletCopySession request for tablet 3481715d755a4277b0905b0832fad4af from peer 623e8dcb58544e558207d8e652dfed5f ({username='slave'} at 127.0.99.129:52947)
I20240503 14:00:28.437372 13795 tablet_copy_service.cc:161] P 66190085d1c04e068d5c0dec9c4bf07d: Beginning new tablet copy session on tablet 3481715d755a4277b0905b0832fad4af from peer 623e8dcb58544e558207d8e652dfed5f at {username='slave'} at 127.0.99.129:52947: session id = 623e8dcb58544e558207d8e652dfed5f-3481715d755a4277b0905b0832fad4af
I20240503 14:00:28.438169 13795 tablet_copy_source_session.cc:205] T 3481715d755a4277b0905b0832fad4af P 66190085d1c04e068d5c0dec9c4bf07d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:28.439283 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:28.439973 22397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3481715d755a4277b0905b0832fad4af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:28.442766 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.460196 22397 tablet_copy_client.cc:783] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:28.460444 22397 tablet_copy_client.cc:647] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:28.463167 22397 tablet_copy_client.cc:523] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:28.467909 22397 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:28.469377 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.475504 22319 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:28.478487 22319 tablet_bootstrap.cc:492] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:00:28.479578 22319 ts_tablet_manager.cc:1367] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.184s user 0.022s sys 0.004s
I20240503 14:00:28.479955 22058 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:28.480212 22058 tablet_bootstrap.cc:492] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:28.480165 22319 raft_consensus.cc:348] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:28.480367 22058 ts_tablet_manager.cc:1367] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.052s user 0.024s sys 0.000s
I20240503 14:00:28.480377 22319 raft_consensus.cc:729] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:00:28.480573 22319 consensus_queue.cc:260] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:28.480765 22058 raft_consensus.cc:348] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:28.480882 22058 raft_consensus.cc:729] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:28.481012 22058 consensus_queue.cc:260] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:28.481331 22319 ts_tablet_manager.cc:1398] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:28.482028 22058 ts_tablet_manager.cc:1398] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:28.482442 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-d1e9155230c74e92b6ee441de713a419 received from {username='slave'} at 127.0.99.136:41157
I20240503 14:00:28.482522 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-d1e9155230c74e92b6ee441de713a419 on tablet d1e9155230c74e92b6ee441de713a419 with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:28.483371 12188 tablet_copy_service.cc:342] P 8b17d706d2344dbaae13cda9b0733143: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-f4f9cb87c70e43dbb0d850c275ebb8a5 received from {username='slave'} at 127.0.99.133:52409
I20240503 14:00:28.483444 12188 tablet_copy_service.cc:433] P 8b17d706d2344dbaae13cda9b0733143: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-f4f9cb87c70e43dbb0d850c275ebb8a5 on tablet f4f9cb87c70e43dbb0d850c275ebb8a5 with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:28.495348 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:28.496181 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:28.497702 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.511780 12429 raft_consensus.cc:1203] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:28.514218 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.523411 13775 consensus_queue.cc:237] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:28.525394 12166 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240503 14:00:28.525772 13711 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): Couldn't send request to peer 623e8dcb58544e558207d8e652dfed5f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:28.525868 22193 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:28.526612 13905 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:28.526997 22193 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:28.527225 22397 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:28.527537 22397 tablet_bootstrap.cc:492] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f: Bootstrap complete.
I20240503 14:00:28.527729 22397 ts_tablet_manager.cc:1367] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.060s user 0.028s sys 0.000s
I20240503 14:00:28.528132 22397 raft_consensus.cc:348] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:28.528229 22397 raft_consensus.cc:729] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:28.528331 22397 consensus_queue.cc:260] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:28.528566 22397 ts_tablet_manager.cc:1398] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:28.529093 13795 tablet_copy_service.cc:342] P 66190085d1c04e068d5c0dec9c4bf07d: Request end of tablet copy session 623e8dcb58544e558207d8e652dfed5f-3481715d755a4277b0905b0832fad4af received from {username='slave'} at 127.0.99.129:52947
I20240503 14:00:28.529160 13795 tablet_copy_service.cc:433] P 66190085d1c04e068d5c0dec9c4bf07d: ending tablet copy session 623e8dcb58544e558207d8e652dfed5f-3481715d755a4277b0905b0832fad4af on tablet 3481715d755a4277b0905b0832fad4af with peer 623e8dcb58544e558207d8e652dfed5f
I20240503 14:00:28.543249 12298 consensus_queue.cc:237] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:28.543810 22419 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:28.544617 14037 raft_consensus.cc:1261] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:28.545029 21772 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:28.545696 14167 raft_consensus.cc:1261] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:28.546682 21772 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:28.546651 13905 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:28.549806 12166 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:28.551080 11895 catalog_manager.cc:5484] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 reported cstate change: config changed from index 8 to 9, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:28.552008 21772 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.554272 14167 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.555377 11890 catalog_manager.cc:5484] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 8 to 9, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.557933 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet c2e9f6eecbec45988b290775277dbddc with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:28.559664 22457 tablet_replica.cc:330] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:28.559846 22457 raft_consensus.cc:2229] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:28.559980 22457 raft_consensus.cc:2258] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:28.559851 14037 raft_consensus.cc:2932] T c2e9f6eecbec45988b290775277dbddc P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.560443 22457 ts_tablet_manager.cc:1875] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:28.562805 22457 ts_tablet_manager.cc:1888] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:00:28.562875 22457 log.cc:1196] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c2e9f6eecbec45988b290775277dbddc
I20240503 14:00:28.563489 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet c2e9f6eecbec45988b290775277dbddc (table rebalance_test_table_1 [id=14df5f9889f146058ff6837fc3ab4239]) successfully deleted
I20240503 14:00:28.589851 13775 consensus_queue.cc:237] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
W20240503 14:00:28.591567 13711 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399): Couldn't send request to peer ca5386f84994496fa5376624391728ee. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:28.591619 13905 raft_consensus.cc:1203] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240503 14:00:28.591727 12166 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:28.592077 22193 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 14:00:28.592859 13711 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): Couldn't send request to peer 623e8dcb58544e558207d8e652dfed5f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:28.594345 13905 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:28.594835 22500 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:28.599680 12298 consensus_queue.cc:237] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:28.600977 13775 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:00:28.601306 12235 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6a89230883347d4a64d96d321247012. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:28.601846 22455 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:28.603029 14167 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:28.603441 22500 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:28.603610 22455 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:28.606549 13905 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:28.610590 11893 catalog_manager.cc:5484] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 reported cstate change: config changed from index 9 to 10, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:28.610843 12166 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:28.617375 22455 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.619583 13775 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.621800 11895 catalog_manager.cc:5484] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.623348 14167 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.678084 12036 consensus_queue.cc:237] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:28.679627 13905 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:00:28.680274 11970 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): Couldn't send request to peer 84c63c6f551949f6a1d7c07760824361. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:28.680881 22552 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:28.682384 13775 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:28.682757 22552 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:28.690248 22552 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:28.690651 12298 consensus_queue.cc:237] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:28.692054 14037 raft_consensus.cc:1261] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:28.692258 13775 raft_consensus.cc:1261] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:00:28.692299 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79b21b1e5292488aad442060b417f3c1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:28.692478 13905 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:28.692590 21772 consensus_queue.cc:1035] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:28.694162 21772 consensus_queue.cc:1035] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:28.696300 11888 catalog_manager.cc:5484] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:28.699780 13757 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:28.712711 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.722134 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:28.722205 13757 raft_consensus.cc:1203] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 14:00:28.726375 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.726910 21772 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:28.727499 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.729475 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:28.734267 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.734764 13775 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:28.735796 14037 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:28.736387 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.741446 11893 catalog_manager.cc:5484] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 7 to 8, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:28.748214 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.749722 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:28.750032 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.773373 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:28.784255 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:28.796283 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.799134 12429 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:28.799456 21738 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:28.805595 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.809993 14167 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:00:28.812299 21772 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:28.814767 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.825829 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:28.872149 12036 consensus_queue.cc:237] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:28.874080 13775 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:28.874316 13905 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:28.874719 22618 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:28.874876 21626 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 14:00:28.875383 11973 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faa0a6ca00d442eaaf9fca7fc1259ec7. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:28.875422 11970 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): Couldn't send request to peer 84c63c6f551949f6a1d7c07760824361. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:28.878775 22618 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.880949 13775 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.882393 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.882763 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.882987 11892 catalog_manager.cc:5484] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 8 to 9, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.884357 13905 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:28.886991 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:28.899255 12036 consensus_queue.cc:237] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:00:28.900466 14036 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:28.900802 13905 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:00:28.901060 21626 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:28.901427 22619 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:28.904174 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.912600 22729 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:28.917744 11893 catalog_manager.cc:5484] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 6 to 7, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:28.919132 13905 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:28.921159 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b4e16598b740cb9d690c2442a3e994 with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33028
I20240503 14:00:28.923350 14036 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:00:28.923553 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:28.923709 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:28.925904 22754 tablet_replica.cc:330] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143: stopping tablet replica
I20240503 14:00:28.926221 22754 raft_consensus.cc:2229] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:28.926563 22754 raft_consensus.cc:2258] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:28.927666 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:28.934363 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:28.934849 22754 ts_tablet_manager.cc:1875] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:28.938369 22754 ts_tablet_manager.cc:1888] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:00:28.938486 22754 log.cc:1196] T a4b4e16598b740cb9d690c2442a3e994 P 8b17d706d2344dbaae13cda9b0733143: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4b4e16598b740cb9d690c2442a3e994
I20240503 14:00:28.939014 11882 catalog_manager.cc:4830] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): tablet a4b4e16598b740cb9d690c2442a3e994 (table rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]) successfully deleted
I20240503 14:00:28.962157 14167 raft_consensus.cc:1203] T d1e9155230c74e92b6ee441de713a419 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 14:00:28.980829 12036 consensus_queue.cc:237] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:28.982386 12166 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:28.982908 13905 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:28.983286 14036 consensus_queue.cc:237] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:28.984146 21626 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:00:28.984619 14167 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Refusing update from remote peer e85b0283a0fd4e3ba81b4170f3cdacb5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:28.984958 12429 raft_consensus.cc:1261] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer e85b0283a0fd4e3ba81b4170f3cdacb5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:28.985186 21449 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:28.985514 21449 consensus_queue.cc:1035] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 14:00:28.985913 11973 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f -> Peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399): Couldn't send request to peer ca5386f84994496fa5376624391728ee. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:28.986070 22619 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:00:28.986616 13973 consensus_peers.cc:485] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 -> Peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): Couldn't send request to peer 623e8dcb58544e558207d8e652dfed5f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:28.990288 21626 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:28.992499 12166 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:28.993549 11893 catalog_manager.cc:5484] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 5 to 6, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:28.993872 13905 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:28.998072 22787 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:29.000125 12429 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:29.002346 14167 raft_consensus.cc:2932] T f5c06806a19e4f4592d26ed4acc651cd P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:29.002442 11893 catalog_manager.cc:5484] T f5c06806a19e4f4592d26ed4acc651cd P e85b0283a0fd4e3ba81b4170f3cdacb5 reported cstate change: config changed from index 8 to 9, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:00:29.020385 12036 raft_consensus.cc:1203] T 3481715d755a4277b0905b0832fad4af P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:00:29.042729 14167 consensus_queue.cc:237] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:29.044199 12166 raft_consensus.cc:1261] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer 1b4a61e8a11440e7a9f4b300d2442f75: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:29.044669 21959 consensus_queue.cc:1035] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:29.046655 21959 raft_consensus.cc:2932] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:29.049914 12166 raft_consensus.cc:2932] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:29.050737 11890 catalog_manager.cc:5484] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 reported cstate change: config changed from index 5 to 7, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:29.054090 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet 211de6f35a844e278e8c1eb1621e7c5f with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33924
I20240503 14:00:29.054380 22457 tablet_replica.cc:330] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:00:29.054534 22457 raft_consensus.cc:2229] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:29.054654 22457 raft_consensus.cc:2258] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:29.055701 22457 ts_tablet_manager.cc:1875] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:29.058022 22457 ts_tablet_manager.cc:1888] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:00:29.058094 22457 log.cc:1196] T 211de6f35a844e278e8c1eb1621e7c5f P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/211de6f35a844e278e8c1eb1621e7c5f
I20240503 14:00:29.058735 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet 211de6f35a844e278e8c1eb1621e7c5f (table rebalance_test_table_3 [id=8e37918da7d2407cbb2fc828ae4e301b]) successfully deleted
I20240503 14:00:29.060851 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:29.063457 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.073496 14167 consensus_queue.cc:237] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:29.075206 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.079994 12166 raft_consensus.cc:1261] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer 1b4a61e8a11440e7a9f4b300d2442f75: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:29.080336 21959 consensus_queue.cc:1035] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:29.084695 21959 raft_consensus.cc:2932] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:29.087430 11881 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 211de6f35a844e278e8c1eb1621e7c5f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:00:29.087798 12166 raft_consensus.cc:2932] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:29.088985 11888 catalog_manager.cc:5484] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 reported cstate change: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } }
I20240503 14:00:29.092424 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:00:29.094008 14104 consensus_peers.cc:485] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 211de6f35a844e278e8c1eb1621e7c5f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:29.104501 22898 ts_tablet_manager.cc:902] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Initiating tablet copy from peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005)
I20240503 14:00:29.105198 22898 tablet_copy_client.cc:308] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37005
I20240503 14:00:29.106566 13795 tablet_copy_service.cc:140] P 66190085d1c04e068d5c0dec9c4bf07d: Received BeginTabletCopySession request for tablet 5e6ca408c4254678b67a74f49a34881b from peer 623e8dcb58544e558207d8e652dfed5f ({username='slave'} at 127.0.99.129:52947)
I20240503 14:00:29.106678 13795 tablet_copy_service.cc:161] P 66190085d1c04e068d5c0dec9c4bf07d: Beginning new tablet copy session on tablet 5e6ca408c4254678b67a74f49a34881b from peer 623e8dcb58544e558207d8e652dfed5f at {username='slave'} at 127.0.99.129:52947: session id = 623e8dcb58544e558207d8e652dfed5f-5e6ca408c4254678b67a74f49a34881b
I20240503 14:00:29.107772 13795 tablet_copy_source_session.cc:205] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:29.109053 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:29.109901 12166 consensus_queue.cc:237] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
W20240503 14:00:29.111722 12101 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): Couldn't send request to peer 84c63c6f551949f6a1d7c07760824361. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c42fee9fadbb4601a7fe32eb6a04e767. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:29.112239 13905 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:29.112731 14167 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:00:29.113111 21739 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:29.113396 21739 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:29.115445 22898 ts_tablet_manager.cc:1875] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:29.117988 22898 ts_tablet_manager.cc:1888] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.5
I20240503 14:00:29.118173 22898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6ca408c4254678b67a74f49a34881b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:29.119290 22898 tablet_copy_client.cc:783] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:29.119423 22898 tablet_copy_client.cc:647] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:29.119760 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:29.127525 22898 tablet_copy_client.cc:523] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:29.128929 22898 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:00:29.130317 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.132390 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:29.145135 12298 consensus_queue.cc:237] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:29.146652 14168 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 14:00:29.146788 12232 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): Couldn't send request to peer 84c63c6f551949f6a1d7c07760824361. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:29.147001 12235 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6a89230883347d4a64d96d321247012. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:29.147133 22519 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:29.147163 13775 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:00:29.147512 21772 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:29.150378 22519 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:29.150692 14167 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:29.151547 13775 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:29.152329 21739 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:29.153489 14167 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:29.153666 11888 catalog_manager.cc:5484] T c42fee9fadbb4601a7fe32eb6a04e767 P 1b4a61e8a11440e7a9f4b300d2442f75 reported cstate change: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:29.154476 11889 catalog_manager.cc:5484] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 8 to 9, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:29.159457 13906 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:29.164453 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:29.189196 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.190495 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.205749 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.207962 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:29.209764 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:29.210532 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:29.232451 22946 ts_tablet_manager.cc:902] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Initiating tablet copy from peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005)
I20240503 14:00:29.233182 22946 tablet_copy_client.cc:308] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37005
I20240503 14:00:29.233655 13795 tablet_copy_service.cc:140] P 66190085d1c04e068d5c0dec9c4bf07d: Received BeginTabletCopySession request for tablet 5e6ca408c4254678b67a74f49a34881b from peer ca5386f84994496fa5376624391728ee ({username='slave'} at 127.0.99.131:46779)
I20240503 14:00:29.233741 13795 tablet_copy_service.cc:161] P 66190085d1c04e068d5c0dec9c4bf07d: Beginning new tablet copy session on tablet 5e6ca408c4254678b67a74f49a34881b from peer ca5386f84994496fa5376624391728ee at {username='slave'} at 127.0.99.131:46779: session id = ca5386f84994496fa5376624391728ee-5e6ca408c4254678b67a74f49a34881b
I20240503 14:00:29.234678 13795 tablet_copy_source_session.cc:205] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:29.234934 14017 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:29.235692 22946 ts_tablet_manager.cc:1875] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:29.243832 22898 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:29.244139 22898 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Bootstrap complete.
I20240503 14:00:29.244321 22898 ts_tablet_manager.cc:1367] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.115s user 0.024s sys 0.001s
I20240503 14:00:29.244750 22898 raft_consensus.cc:348] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:29.244880 22898 raft_consensus.cc:729] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:00:29.245036 22898 consensus_queue.cc:260] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:29.245370 22898 ts_tablet_manager.cc:1398] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:29.246958 13795 tablet_copy_service.cc:342] P 66190085d1c04e068d5c0dec9c4bf07d: Request end of tablet copy session 623e8dcb58544e558207d8e652dfed5f-5e6ca408c4254678b67a74f49a34881b received from {username='slave'} at 127.0.99.129:52947
I20240503 14:00:29.247027 13795 tablet_copy_service.cc:433] P 66190085d1c04e068d5c0dec9c4bf07d: ending tablet copy session 623e8dcb58544e558207d8e652dfed5f-5e6ca408c4254678b67a74f49a34881b on tablet 5e6ca408c4254678b67a74f49a34881b with peer 623e8dcb58544e558207d8e652dfed5f
I20240503 14:00:29.250068 22948 ts_tablet_manager.cc:902] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:29.250604 22946 ts_tablet_manager.cc:1888] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 14:00:29.250800 22946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6ca408c4254678b67a74f49a34881b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:29.252053 22948 tablet_copy_client.cc:308] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:29.252519 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet 79b21b1e5292488aad442060b417f3c1 from peer 1b4a61e8a11440e7a9f4b300d2442f75 ({username='slave'} at 127.0.99.136:41157)
I20240503 14:00:29.252605 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet 79b21b1e5292488aad442060b417f3c1 from peer 1b4a61e8a11440e7a9f4b300d2442f75 at {username='slave'} at 127.0.99.136:41157: session id = 1b4a61e8a11440e7a9f4b300d2442f75-79b21b1e5292488aad442060b417f3c1
I20240503 14:00:29.253386 12319 tablet_copy_source_session.cc:205] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:29.254659 22948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b21b1e5292488aad442060b417f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:29.254627 22946 tablet_copy_client.cc:783] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:29.254810 22946 tablet_copy_client.cc:647] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:29.256273 22946 tablet_copy_client.cc:523] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:29.260720 22983 ts_tablet_manager.cc:902] T faa0a6ca00d442eaaf9fca7fc1259ec7 P e85b0283a0fd4e3ba81b4170f3cdacb5: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
F20240503 14:00:29.260885 22983 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.queue_time_us, tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.queue_time_us) (tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.run_cpu_time_us, tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.run_cpu_time_us) (tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.run_wall_time_us, tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.run_wall_time_us) (tablet-download-pool-11a8c270532947ba9b505aace44b56f6.queue_time_us, tablet-download-pool-11a8c270532947ba9b505aace44b56f6.queue_time_us) (tablet-download-pool-11a8c270532947ba9b505aace44b56f6.run_cpu_time_us, tablet-download-pool-11a8c270532947ba9b505aace44b56f6.run_cpu_time_us) (tablet-download-pool-11a8c270532947ba9b505aace44b56f6.run_wall_time_us, tablet-download-pool-11a8c270532947ba9b505aace44b56f6.run_wall_time_us) (tablet-download-pool-4224ca4d93ce4784b52a7550468ce1b1.queue_time_us, tablet-download-pool-4224ca4d93ce4784b52a7550468ce1b1.queue_time_us) (tablet-download-pool-4224ca4d93ce4784b52a7550468ce1b1.run_cpu_time_us, tablet-download-pool-4224ca4d93ce4784b52a7550468ce1b1.run_cpu_time_us) (tablet-download-pool-4224ca4d93ce4784b52a7550468ce1b1.run_wall_time_us, tablet-download-pool-4224ca4d93ce4784b52a7550468ce1b1.run_wall_time_us) (tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.queue_time_us, tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.queue_time_us) (tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.run_cpu_time_us, tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.run_cpu_time_us) (tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.run_wall_time_us, tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.run_wall_time_us) (tablet-download-pool-79b21b1e5292488aad442060b417f3c1.queue_time_us, tablet-download-pool-79b21b1e5292488aad442060b417f3c1.queue_time_us) (tablet-download-pool-79b21b1e5292488aad442060b417f3c1.run_cpu_time_us, tablet-download-pool-79b21b1e5292488aad442060b417f3c1.run_cpu_time_us) (tablet-download-pool-79b21b1e5292488aad442060b417f3c1.run_wall_time_us, tablet-download-pool-79b21b1e5292488aad442060b417f3c1.run_wall_time_us) (tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.queue_time_us, tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.queue_time_us) (tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_cpu_time_us, tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_cpu_time_us) (tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_wall_time_us, tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_wall_time_us) (tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.queue_time_us, tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.queue_time_us) (tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.run_cpu_time_us, tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.run_cpu_time_us) (tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.run_wall_time_us, tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.run_wall_time_us) (tablet-download-pool-adf2c44328e244779955e712b99f8331.queue_time_us, tablet-download-pool-adf2c44328e244779955e712b99f8331.queue_time_us) (tablet-download-pool-adf2c44328e244779955e712b99f8331.run_cpu_time_us, tablet-download-pool-adf2c44328e244779955e712b99f8331.run_cpu_time_us) (tablet-download-pool-adf2c44328e244779955e712b99f8331.run_wall_time_us, tablet-download-pool-adf2c44328e244779955e712b99f8331.run_wall_time_us) (tablet-download-pool-b031f09b97664d1a94c14f0661181899.queue_time_us, tablet-download-pool-b031f09b97664d1a94c14f0661181899.queue_time_us) (tablet-download-pool-b031f09b97664d1a94c14f0661181899.run_cpu_time_us, tablet-download-pool-b031f09b97664d1a94c14f0661181899.run_cpu_time_us) (tablet-download-pool-b031f09b97664d1a94c14f0661181899.run_wall_time_us, tablet-download-pool-b031f09b97664d1a94c14f0661181899.run_wall_time_us) (tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.queue_time_us, tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.queue_time_us) (tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.run_cpu_time_us, tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.run_cpu_time_us) (tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.run_wall_time_us, tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.run_wall_time_us) (tablet-download-pool-d1e9155230c74e92b6ee441de713a419.queue_time_us, tablet-download-pool-d1e9155230c74e92b6ee441de713a419.queue_time_us) (tablet-download-pool-d1e9155230c74e92b6ee441de713a419.run_cpu_time_us, tablet-download-pool-d1e9155230c74e92b6ee441de713a419.run_cpu_time_us) (tablet-download-pool-d1e9155230c74e92b6ee441de713a419.run_wall_time_us, tablet-download-pool-d1e9155230c74e92b6ee441de713a419.run_wall_time_us) (tablet-download-pool-d837845c6f644bca83315c9f7c45246d.queue_time_us, tablet-download-pool-d837845c6f644bca83315c9f7c45246d.queue_time_us) (tablet-download-pool-d837845c6f644bca83315c9f7c45246d.run_cpu_time_us, tablet-download-pool-d837845c6f644bca83315c9f7c45246d.run_cpu_time_us) (tablet-download-pool-d837845c6f644bca83315c9f7c45246d.run_wall_time_us, tablet-download-pool-d837845c6f644bca83315c9f7c45246d.run_wall_time_us) (tablet-download-pool-f4f9cb87c70e43dbb0d850c275ebb8a5.queue_time_us, tablet-download-pool-f4f9cb87c70e43dbb0d850c275ebb8a5.queue_time_us) (tablet-download-pool-f4f9cb87c70e43dbb0d850c275ebb8a5.run_cpu_time_us, tablet-download-pool-f4f9cb87c70e43dbb0d850c275ebb8a5.run_cpu_time_us) (tablet-download-pool-f4f9cb87c70e43dbb0d850c275ebb8a5.run_wall_time_us, tablet-download-pool-f4f9cb87c70e43dbb0d850c275ebb8a5.run_wall_time_us) (tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.queue_time_us, tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.queue_time_us) (tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.run_cpu_time_us, tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.run_cpu_time_us) (tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.run_wall_time_us, tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.run_wall_time_us) (tablet-download-pool-faa0a6ca00d442eaaf9fca7fc1259ec7.queue_time_us, tablet-download-pool-faa0a6ca00d442eaaf9fca7fc1259ec7.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f2f483ecdcd google::LogMessage::Fail() at ??:0
I20240503 14:00:29.265197 22948 tablet_copy_client.cc:783] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:29.265363 22948 tablet_copy_client.cc:647] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:29.265765 22946 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Bootstrap starting.
@ 0x7f2f483f0b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f2f483ec7cc google::LogMessage::Flush() at ??:0
I20240503 14:00:29.268968 22948 tablet_copy_client.cc:523] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f2f483edf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2f49856044 kudu::TraceMetrics::InternName() at ??:0
I20240503 14:00:29.276065 22948 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap starting.
@ 0x7f2f4983af98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f2f49838efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f2f4c3eb77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f2f4c3f4a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 14:00:29.287082 23012 ts_tablet_manager.cc:902] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:29.287501 23012 tablet_copy_client.cc:308] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:29.287932 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet faa0a6ca00d442eaaf9fca7fc1259ec7 from peer 84c63c6f551949f6a1d7c07760824361 ({username='slave'} at 127.0.99.132:42449)
I20240503 14:00:29.288013 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet faa0a6ca00d442eaaf9fca7fc1259ec7 from peer 84c63c6f551949f6a1d7c07760824361 at {username='slave'} at 127.0.99.132:42449: session id = 84c63c6f551949f6a1d7c07760824361-faa0a6ca00d442eaaf9fca7fc1259ec7
I20240503 14:00:29.288798 12057 tablet_copy_source_session.cc:205] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:29.290067 12036 consensus_queue.cc:237] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:29.290230 23012 ts_tablet_manager.cc:1875] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7f2f4c47576f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 14:00:29.291661 12166 raft_consensus.cc:1261] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:00:29.293493 22618 consensus_queue.cc:1035] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7f2f4c4894f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 14:00:29.294162 14036 raft_consensus.cc:1261] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 14:00:29.294656 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d837845c6f644bca83315c9f7c45246d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:29.295589 22618 consensus_queue.cc:1035] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7f2f4c474226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f2f4c483c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56466da2b75c std::function<>::operator()() at ??:0
@ 0x7f2f4983de06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2f4983e6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f2f4984003b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56466da2b75c std::function<>::operator()() at ??:0
I20240503 14:00:29.309127 22618 raft_consensus.cc:2932] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:29.309358 23012 ts_tablet_manager.cc:1888] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 14:00:29.309584 23012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa0a6ca00d442eaaf9fca7fc1259ec7. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f2f4982d807 kudu::Thread::SuperviseThread() at ??:0
I20240503 14:00:29.311725 14036 raft_consensus.cc:2932] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:29.313514 12166 raft_consensus.cc:2932] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
@ 0x7f2f4a06b6db start_thread at ??:0
@ 0x7f2f475a371f clone at ??:0
I20240503 14:00:29.316283 11894 catalog_manager.cc:5484] T d837845c6f644bca83315c9f7c45246d P e85b0283a0fd4e3ba81b4170f3cdacb5 reported cstate change: config changed from index 5 to 6, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:00:29.327034 23012 tablet_copy_client.cc:783] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:29.327337 23012 tablet_copy_client.cc:647] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:29.329773 23012 tablet_copy_client.cc:523] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:29.331883 23012 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:29.379737 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:29.387598 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:29.396226 22948 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:29.396652 22948 tablet_bootstrap.cc:492] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75: Bootstrap complete.
I20240503 14:00:29.396701 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.396852 22948 ts_tablet_manager.cc:1367] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent bootstrapping tablet: real 0.121s user 0.025s sys 0.004s
I20240503 14:00:29.397809 22948 raft_consensus.cc:348] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:29.397926 22948 raft_consensus.cc:729] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4a61e8a11440e7a9f4b300d2442f75, State: Initialized, Role: FOLLOWER
I20240503 14:00:29.398115 22948 consensus_queue.cc:260] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:00:29.398561 22948 ts_tablet_manager.cc:1398] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:00:29.399197 12319 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-79b21b1e5292488aad442060b417f3c1 received from {username='slave'} at 127.0.99.136:41157
I20240503 14:00:29.399287 12319 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 1b4a61e8a11440e7a9f4b300d2442f75-79b21b1e5292488aad442060b417f3c1 on tablet 79b21b1e5292488aad442060b417f3c1 with peer 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:29.420380 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:29.432193 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.449887 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:29.468669 12298 consensus_queue.cc:237] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:29.470291 14167 raft_consensus.cc:1261] T c2e9f6eecbec45988b290775277dbddc P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 14:00:29.470600 12232 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): Couldn't send request to peer 84c63c6f551949f6a1d7c07760824361. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:29.470835 21772 consensus_queue.cc:1035] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:29.481273 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.488413 22946 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:29.488744 22946 tablet_bootstrap.cc:492] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Bootstrap complete.
I20240503 14:00:29.488943 22946 ts_tablet_manager.cc:1367] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.223s user 0.027s sys 0.000s
I20240503 14:00:29.489562 22946 raft_consensus.cc:348] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:29.489782 22946 raft_consensus.cc:729] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:29.489943 22946 consensus_queue.cc:260] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:29.490376 22946 ts_tablet_manager.cc:1398] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 14:00:29.491276 13795 tablet_copy_service.cc:342] P 66190085d1c04e068d5c0dec9c4bf07d: Request end of tablet copy session ca5386f84994496fa5376624391728ee-5e6ca408c4254678b67a74f49a34881b received from {username='slave'} at 127.0.99.131:46779
I20240503 14:00:29.491360 13795 tablet_copy_service.cc:433] P 66190085d1c04e068d5c0dec9c4bf07d: ending tablet copy session ca5386f84994496fa5376624391728ee-5e6ca408c4254678b67a74f49a34881b on tablet 5e6ca408c4254678b67a74f49a34881b with peer ca5386f84994496fa5376624391728ee
I20240503 14:00:29.501631 23012 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:29.501946 23012 tablet_bootstrap.cc:492] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Bootstrap complete.
I20240503 14:00:29.502133 23012 ts_tablet_manager.cc:1367] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.170s user 0.020s sys 0.006s
I20240503 14:00:29.502655 23012 raft_consensus.cc:348] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:29.502789 23012 raft_consensus.cc:729] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:29.502949 23012 consensus_queue.cc:260] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:29.514178 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:29.516657 23012 ts_tablet_manager.cc:1398] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.014s user 0.003s sys 0.001s
I20240503 14:00:29.518828 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session 84c63c6f551949f6a1d7c07760824361-faa0a6ca00d442eaaf9fca7fc1259ec7 received from {username='slave'} at 127.0.99.132:42449
I20240503 14:00:29.518930 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session 84c63c6f551949f6a1d7c07760824361-faa0a6ca00d442eaaf9fca7fc1259ec7 on tablet faa0a6ca00d442eaaf9fca7fc1259ec7 with peer 84c63c6f551949f6a1d7c07760824361
I20240503 14:00:29.520349 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:29.521364 23104 ts_tablet_manager.cc:902] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361: Initiating tablet copy from peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:00:29.521945 23104 tablet_copy_client.cc:308] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36263
I20240503 14:00:29.522466 12188 tablet_copy_service.cc:140] P 8b17d706d2344dbaae13cda9b0733143: Received BeginTabletCopySession request for tablet c42fee9fadbb4601a7fe32eb6a04e767 from peer 84c63c6f551949f6a1d7c07760824361 ({username='slave'} at 127.0.99.132:45565)
I20240503 14:00:29.522572 12188 tablet_copy_service.cc:161] P 8b17d706d2344dbaae13cda9b0733143: Beginning new tablet copy session on tablet c42fee9fadbb4601a7fe32eb6a04e767 from peer 84c63c6f551949f6a1d7c07760824361 at {username='slave'} at 127.0.99.132:45565: session id = 84c63c6f551949f6a1d7c07760824361-c42fee9fadbb4601a7fe32eb6a04e767
I20240503 14:00:29.523558 12188 tablet_copy_source_session.cc:205] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:29.533278 23104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42fee9fadbb4601a7fe32eb6a04e767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:29.536628 23104 tablet_copy_client.cc:783] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:29.538224 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.539116 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.540234 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.543077 12036 raft_consensus.cc:1203] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240503 14:00:29.559568 23104 tablet_copy_client.cc:647] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:29.561666 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:29.565959 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:29.567297 23104 tablet_copy_client.cc:523] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:29.569409 23104 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:29.583427 22946 ts_tablet_manager.cc:902] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:29.583868 22946 tablet_copy_client.cc:308] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:00:29.584255 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet 2089eeda1e0f4ff5926584d472e6631d from peer ca5386f84994496fa5376624391728ee ({username='slave'} at 127.0.99.131:56133)
I20240503 14:00:29.584342 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet 2089eeda1e0f4ff5926584d472e6631d from peer ca5386f84994496fa5376624391728ee at {username='slave'} at 127.0.99.131:56133: session id = ca5386f84994496fa5376624391728ee-2089eeda1e0f4ff5926584d472e6631d
I20240503 14:00:29.585220 12057 tablet_copy_source_session.cc:205] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:29.586030 22946 ts_tablet_manager.cc:1875] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:29.588397 22946 ts_tablet_manager.cc:1888] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 14:00:29.588544 22946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2089eeda1e0f4ff5926584d472e6631d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:29.592644 22946 tablet_copy_client.cc:783] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:29.593029 22946 tablet_copy_client.cc:647] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:29.593784 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.594007 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:29.615471 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.623278 22946 tablet_copy_client.cc:523] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:29.628086 22946 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:00:29.631060 23104 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:29.631345 23104 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361: Bootstrap complete.
I20240503 14:00:29.631515 23104 ts_tablet_manager.cc:1367] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.062s user 0.025s sys 0.004s
I20240503 14:00:29.631927 23104 raft_consensus.cc:348] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:29.632018 23104 raft_consensus.cc:729] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:29.632123 23104 consensus_queue.cc:260] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:29.632381 23104 ts_tablet_manager.cc:1398] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:29.632817 12188 tablet_copy_service.cc:342] P 8b17d706d2344dbaae13cda9b0733143: Request end of tablet copy session 84c63c6f551949f6a1d7c07760824361-c42fee9fadbb4601a7fe32eb6a04e767 received from {username='slave'} at 127.0.99.132:45565
I20240503 14:00:29.632913 12188 tablet_copy_service.cc:433] P 8b17d706d2344dbaae13cda9b0733143: ending tablet copy session 84c63c6f551949f6a1d7c07760824361-c42fee9fadbb4601a7fe32eb6a04e767 on tablet c42fee9fadbb4601a7fe32eb6a04e767 with peer 84c63c6f551949f6a1d7c07760824361
I20240503 14:00:29.637616 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:29.660187 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:29.663033 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.698258 22946 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:29.698663 22946 tablet_bootstrap.cc:492] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Bootstrap complete.
I20240503 14:00:29.698853 22946 ts_tablet_manager.cc:1367] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.071s user 0.017s sys 0.007s
I20240503 14:00:29.700940 22946 raft_consensus.cc:348] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:29.701069 22946 raft_consensus.cc:729] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:00:29.701215 22946 consensus_queue.cc:260] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:29.701567 22946 ts_tablet_manager.cc:1398] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:00:29.702138 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session ca5386f84994496fa5376624391728ee-2089eeda1e0f4ff5926584d472e6631d received from {username='slave'} at 127.0.99.131:56133
I20240503 14:00:29.702220 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session ca5386f84994496fa5376624391728ee-2089eeda1e0f4ff5926584d472e6631d on tablet 2089eeda1e0f4ff5926584d472e6631d with peer ca5386f84994496fa5376624391728ee
I20240503 14:00:29.719951 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:29.723527 22948 ts_tablet_manager.cc:902] T d837845c6f644bca83315c9f7c45246d P 1b4a61e8a11440e7a9f4b300d2442f75: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
F20240503 14:00:29.723743 22948 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-11a8c270532947ba9b505aace44b56f6.queue_time_us, tablet-download-pool-11a8c270532947ba9b505aace44b56f6.queue_time_us) (tablet-download-pool-11a8c270532947ba9b505aace44b56f6.run_cpu_time_us, tablet-download-pool-11a8c270532947ba9b505aace44b56f6.run_cpu_time_us) (tablet-download-pool-11a8c270532947ba9b505aace44b56f6.run_wall_time_us, tablet-download-pool-11a8c270532947ba9b505aace44b56f6.run_wall_time_us) (tablet-download-pool-211de6f35a844e278e8c1eb1621e7c5f.queue_time_us, tablet-download-pool-211de6f35a844e278e8c1eb1621e7c5f.queue_time_us) (tablet-download-pool-211de6f35a844e278e8c1eb1621e7c5f.run_cpu_time_us, tablet-download-pool-211de6f35a844e278e8c1eb1621e7c5f.run_cpu_time_us) (tablet-download-pool-211de6f35a844e278e8c1eb1621e7c5f.run_wall_time_us, tablet-download-pool-211de6f35a844e278e8c1eb1621e7c5f.run_wall_time_us) (tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.queue_time_us, tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.queue_time_us) (tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.run_cpu_time_us, tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.run_cpu_time_us) (tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.run_wall_time_us, tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.run_wall_time_us) (tablet-download-pool-79b21b1e5292488aad442060b417f3c1.queue_time_us, tablet-download-pool-79b21b1e5292488aad442060b417f3c1.queue_time_us) (tablet-download-pool-79b21b1e5292488aad442060b417f3c1.run_cpu_time_us, tablet-download-pool-79b21b1e5292488aad442060b417f3c1.run_cpu_time_us) (tablet-download-pool-79b21b1e5292488aad442060b417f3c1.run_wall_time_us, tablet-download-pool-79b21b1e5292488aad442060b417f3c1.run_wall_time_us) (tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.queue_time_us, tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.queue_time_us) (tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_cpu_time_us, tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_cpu_time_us) (tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_wall_time_us, tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_wall_time_us) (tablet-download-pool-a6a89230883347d4a64d96d321247012.queue_time_us, tablet-download-pool-a6a89230883347d4a64d96d321247012.queue_time_us) (tablet-download-pool-a6a89230883347d4a64d96d321247012.run_cpu_time_us, tablet-download-pool-a6a89230883347d4a64d96d321247012.run_cpu_time_us) (tablet-download-pool-a6a89230883347d4a64d96d321247012.run_wall_time_us, tablet-download-pool-a6a89230883347d4a64d96d321247012.run_wall_time_us) (tablet-download-pool-b031f09b97664d1a94c14f0661181899.queue_time_us, tablet-download-pool-b031f09b97664d1a94c14f0661181899.queue_time_us) (tablet-download-pool-b031f09b97664d1a94c14f0661181899.run_cpu_time_us, tablet-download-pool-b031f09b97664d1a94c14f0661181899.run_cpu_time_us) (tablet-download-pool-b031f09b97664d1a94c14f0661181899.run_wall_time_us, tablet-download-pool-b031f09b97664d1a94c14f0661181899.run_wall_time_us) (tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.queue_time_us, tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.queue_time_us) (tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.run_cpu_time_us, tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.run_cpu_time_us) (tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.run_wall_time_us, tablet-download-pool-c2e9f6eecbec45988b290775277dbddc.run_wall_time_us) (tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.queue_time_us, tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.queue_time_us) (tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.run_cpu_time_us, tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.run_cpu_time_us) (tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.run_wall_time_us, tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.run_wall_time_us) (tablet-download-pool-c8933689df2c45b59a9cd16024ddb7d9.queue_time_us, tablet-download-pool-c8933689df2c45b59a9cd16024ddb7d9.queue_time_us) (tablet-download-pool-c8933689df2c45b59a9cd16024ddb7d9.run_cpu_time_us, tablet-download-pool-c8933689df2c45b59a9cd16024ddb7d9.run_cpu_time_us) (tablet-download-pool-c8933689df2c45b59a9cd16024ddb7d9.run_wall_time_us, tablet-download-pool-c8933689df2c45b59a9cd16024ddb7d9.run_wall_time_us) (tablet-download-pool-d1e9155230c74e92b6ee441de713a419.queue_time_us, tablet-download-pool-d1e9155230c74e92b6ee441de713a419.queue_time_us) (tablet-download-pool-d1e9155230c74e92b6ee441de713a419.run_cpu_time_us, tablet-download-pool-d1e9155230c74e92b6ee441de713a419.run_cpu_time_us) (tablet-download-pool-d1e9155230c74e92b6ee441de713a419.run_wall_time_us, tablet-download-pool-d1e9155230c74e92b6ee441de713a419.run_wall_time_us) (tablet-download-pool-d837845c6f644bca83315c9f7c45246d.queue_time_us, tablet-download-pool-d837845c6f644bca83315c9f7c45246d.queue_time_us) (tablet-download-pool-df13ec098dc548afac8c67195d25e7e8.queue_time_us, tablet-download-pool-df13ec098dc548afac8c67195d25e7e8.queue_time_us) (tablet-download-pool-df13ec098dc548afac8c67195d25e7e8.run_cpu_time_us, tablet-download-pool-df13ec098dc548afac8c67195d25e7e8.run_cpu_time_us) (tablet-download-pool-df13ec098dc548afac8c67195d25e7e8.run_wall_time_us, tablet-download-pool-df13ec098dc548afac8c67195d25e7e8.run_wall_time_us) (tablet-download-pool-e92d8f55591346158b9583feb958a8ca.queue_time_us, tablet-download-pool-e92d8f55591346158b9583feb958a8ca.queue_time_us) (tablet-download-pool-e92d8f55591346158b9583feb958a8ca.run_cpu_time_us, tablet-download-pool-e92d8f55591346158b9583feb958a8ca.run_cpu_time_us) (tablet-download-pool-e92d8f55591346158b9583feb958a8ca.run_wall_time_us, tablet-download-pool-e92d8f55591346158b9583feb958a8ca.run_wall_time_us) (tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.queue_time_us, tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.queue_time_us) (tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.run_cpu_time_us, tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.run_cpu_time_us) (tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.run_wall_time_us, tablet-download-pool-f5c06806a19e4f4592d26ed4acc651cd.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
W20240503 14:00:29.724488 14104 connection.cc:535] client connection to 127.0.99.135:46845 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:29.724633 14104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:29.725788 11880 connection.cc:535] server connection from 127.0.99.135:36225 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7f0f4ba96dcd google::LogMessage::Fail() at ??:0
W20240503 14:00:29.727353 11973 connection.cc:535] client connection to 127.0.99.135:46845 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:29.727468 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:29.728243 12104 connection.cc:535] client connection to 127.0.99.135:46845 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:29.728351 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:29.728749 12235 connection.cc:535] client connection to 127.0.99.135:46845 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:29.728842 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7f0f4ba9ab93 google::LogMessage::SendToLog() at ??:0
W20240503 14:00:29.732614 14104 consensus_peers.cc:462] T 211de6f35a844e278e8c1eb1621e7c5f P 1b4a61e8a11440e7a9f4b300d2442f75 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
@ 0x7f0f4ba967cc google::LogMessage::Flush() at ??:0
@ 0x7f0f4ba97f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
W20240503 14:00:29.737990 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:29.740036 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:29.740109 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:29.740142 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:29.745672 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:29.745685 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:29.745748 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:29.745785 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:29.745815 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:29.745842 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:29.745900 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:29.745927 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f0f4cf00044 kudu::TraceMetrics::InternName() at ??:0
I20240503 14:00:29.746454 23104 ts_tablet_manager.cc:902] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:29.746863 23104 tablet_copy_client.cc:308] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:29.747421 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet a6a89230883347d4a64d96d321247012 from peer 84c63c6f551949f6a1d7c07760824361 ({username='slave'} at 127.0.99.132:33737)
I20240503 14:00:29.747507 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet a6a89230883347d4a64d96d321247012 from peer 84c63c6f551949f6a1d7c07760824361 at {username='slave'} at 127.0.99.132:33737: session id = 84c63c6f551949f6a1d7c07760824361-a6a89230883347d4a64d96d321247012
I20240503 14:00:29.748427 12319 tablet_copy_source_session.cc:205] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f0f4cee4f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240503 14:00:29.749598 23104 ts_tablet_manager.cc:1875] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:29.751725 23104 ts_tablet_manager.cc:1888] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 14:00:29.751885 23104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6a89230883347d4a64d96d321247012. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:29.753105 23104 tablet_copy_client.cc:783] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:29.753248 23104 tablet_copy_client.cc:647] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:29.754493 14148 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:00:29.754695 23104 tablet_copy_client.cc:523] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:29.755856 23104 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
@ 0x7f0f4cee2efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 14:00:29.759589 12298 raft_consensus.cc:1203] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240503 14:00:29.760870 14167 raft_consensus.cc:1203] T 79b21b1e5292488aad442060b417f3c1 P 1b4a61e8a11440e7a9f4b300d2442f75 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:29.761930 12429 raft_consensus.cc:1203] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 14:00:29.764127 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7f0f4fa9577a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f0f4fa9ea45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
W20240503 14:00:29.788534 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:29.790788 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:29.792941 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
@ 0x7f0f4fb1f76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 14:00:29.795498 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:00:29.796074 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f0f4fb334f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f0f4fb1e226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f0f4fb2dc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55983cb9e75c std::function<>::operator()() at ??:0
@ 0x7f0f4cee7e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0f4cee86ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f0f4ceea03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55983cb9e75c std::function<>::operator()() at ??:0
I20240503 14:00:29.824019 14219 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:00:14.040737 14224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:14.041265 14224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:14.045680 14224 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:00:15.210942 14224 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:00:15.815340 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.815491 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.815567 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.815604 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.815691 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.815735 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.815789 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.815833 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.815894 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.815945 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.815984 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.816022 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to serW20240503 14:00:29.847913 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
@ 0x7f0f4ced7807 kudu::Thread::SuperviseThread() at ??:0
W20240503 14:00:29.872663 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f0f4d7156db start_thread at ??:0
@ 0x7f0f4ac4d71f clone at ??:0
W20240503 14:00:29.826968 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:29.912981 12366 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.135:46845, user_credentials={real_user=slave}} blocked reactor thread for 85990.2us
I20240503 14:00:29.831390 23104 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:29.864385 23012 ts_tablet_manager.cc:902] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:00:29.913800 23104 tablet_bootstrap.cc:492] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Bootstrap complete.
I20240503 14:00:29.830924 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.914026 23104 ts_tablet_manager.cc:1367] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.158s user 0.024s sys 0.003s
ver b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.816107 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.816157 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.816192 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:15.873833 14224 rebalancer_tool.cc:1084] tablet f5c06806a19e4f4592d26ed4acc651cd: '623e8dcb58544e558207d8e652dfed5f' -> 'b67f337e10994234baedbf0b19c13513' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 14:00:15.873947 14224 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:16.195442 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.195708 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.195876 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.195919 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.195959 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.196008 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.196040 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.196116 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.196149 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.196280 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.196329 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.197484 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.197567 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.197623 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.197649 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.197690 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.197726 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.197767 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.197803 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.197845 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:16.273167 14224 rebalancer_tool.cc:1074] tablet 6c4f3bc64e7145e2b692beab730d1496: '623e8dcb58544e558207d8e652dfed5f' -> 'b67f337e10994234baedbf0b19c13513' move scheduled
I20240503 14:00:16.326062 14224 rebalancer_tool.cc:1074] tablet 7d2e1517a8ac4041bfb4b90634cb0fa6: '84c63c6f551949f6a1d7c07760824361' -> '66190085d1c04e068d5c0dec9c4bf07d' move scheduled
I20240503 14:00:16.462446 14224 rebalancer_tool.cc:1074] tablet c8933689df2c45b59a9cd16024ddb7d9: '8b17d706d2344dbaae13cda9b0733143' -> '1b4a61e8a11440e7a9f4b300d2442f75' move scheduled
I20240503 14:00:16.492883 14224 rebalancer_tool.cc:1084] tablet d837845c6f644bca83315c9f7c45246d: 'ca5386f84994496fa5376624391728ee' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move ignored: Illegal state: tablet d837845c6f644bca83315c9f7c45246d: replica e85b0283a0fd4e3ba81b4170f3cdacb5 is already present
I20240503 14:00:16.492976 14224 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:17.124963 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.125151 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.125264 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.125308 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.125336 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.125375 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.125412 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.125450 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.125489 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.125532 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.125571 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.125612 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.125654 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.125705 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.125741 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.125779 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:17.200117 14224 rebalancer_tool.cc:1074] tablet 2089eeda1e0f4ff5926584d472e6631d: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:17.281780 14224 rebalancer_tool.cc:1084] tablet 0e442bb664f44887aa8afdb62e1d97fc: '84c63c6f551949f6a1d7c07760824361' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 14:00:17.281867 14224 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:17.669992 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.670159 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.670929 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.671020 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.671067 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.671141 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.671182 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.671221 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.671306 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:18.024363 14224 rebalancer_tool.cc:1074] tablet adf2c44328e244779955e712b99f8331: '623e8dcb58544e558207d8e652dfed5f' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move scheduled
I20240503 14:00:18.040267 14224 rebalancer_tool.cc:1074] tablet f4f9cb87c70e43dbb0d850c275ebb8a5: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:18.063040 14224 rebalancer_tool.cc:1074] tablet a6a89230883347d4a64d96d321247012: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:18.086246 14224 rebalancer_tool.cc:1084] tablet df13ec098dc548afac8c67195d25e7e8: '84c63c6f551949f6a1d7c07760824361' -> '1b4a61e8a11440e7a9f4b300d2442f75' move ignored: Illegal state: tablet df13ec098dc548afac8c67195d25e7e8: replica 1b4a61e8a11440e7a9f4b300d2442f75 is already present
I20240503 14:00:18.086344 14224 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:18.644963 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:18.645066 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:18.645108 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:18.645149 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:18.645236 14224 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:18.851908 14224 rebalancer_tool.cc:1074] tablet faa0a6ca00d442eaaf9fca7fc1259ec7: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:18.864315 14224 rebalancer_tool.cc:1074] tablet a0106f7a40cb467eb4738890ac4e22d4: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:21.806629 14224 rebalancer_tool.cc:1136] tablet f4f9cb87c70e43dbb0d850c275ebb8a5: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:22.971952 14224 rebalancer_tool.cc:1136] tablet adf2c44328e244779955e712b99f8331: '623e8dcb58544e558207d8e652dfed5f' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move completed: OK
I20240503 14:00:22.998651 14224 rebalancer_tool.cc:1074] tablet 4a608f683c4f41f3bbdfc9b4c219ce49: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:23.011936 14224 rebalancer_tool.cc:1136] tablet faa0a6ca00d442eaaf9fca7fc1259ec7: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:23.024724 14224 rebalancer_tool.cc:1136] tablet 4a608f683c4f41f3bbdfc9b4c219ce49: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:23.048708 14224 rebalancer_tool.cc:1136] tablet a6a89230883347d4a64d96d321247012: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
I20240503 14:00:24.204073 14224 rebalancer_tool.cc:1136] tablet 2089eeda1e0f4ff5926584d472e6631d: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:24.212673 14224 rebalancer_tool.cc:1136] tablet 7d2e1517a8ac4041bfb4b90634cb0fa6: '84c63c6f551949f6a1d7c07760824361' -> '66190085d1c04e068d5c0dec9c4bf07d' move completed: OK
I20240503 14:00:24.219622 14224 rebalancer_tool.cc:1136] tablet c8933689df2c45b59a9cd16024ddb7d9: '8b17d706d2344dbaae13cda9b0733143' -> '1b4a61e8a11440e7a9f4b300d2442f75' move completed: OK
I20240503 14:00:24.226163 14224 rebalancer_tool.cc:1074] tablet d837845c6f644bca83315c9f7c45246d: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:24.236480 14224 rebalancer_tool.cc:1074] tablet a4b4e16598b740cb9d690c2442a3e994: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:24.257213 14224 rebalancer_tool.cc:1074] tablet 79b21b1e5292488aad442060b417f3c1: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:24.274160 14224 rebalancer_tool.cc:1074] tablet 3481715d755a4277b0905b0832fad4af: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:24.288300 14224 rebalancer_tool.cc:1136] tablet 79b21b1e5292488aad442060b417f3c1: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:24.302129 14224 rebalancer_tool.cc:1136] tablet a4b4e16598b740cb9d690c2442a3e994: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:24.326395 14224 rebalancer_tool.cc:1136] tablet d837845c6f644bca83315c9f7c45246d: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:24.360370 14224 rebalancer_tool.cc:1136] tablet 6c4f3bc64e7145e2b692beab730d1496: '623e8dcb58544e558207d8e652dfed5f' -> 'b67f337e10994234baedbf0b19c13513' move completed: OK
I20240503 14:00:24.483474 14224 rebalancer_tool.cc:1136] tablet 3481715d755a4277b0905b0832fad4af: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
I20240503 14:00:24.954588 14224 rebalancer_tool.cc:1074] tablet 0e442bb664f44887aa8afdb62e1d97fc: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:25.011736 14224 rebalancer_tool.cc:1074] tablet 5e6ca408c4254678b67a74f49a34881b: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:25.068487 14224 rebalancer_tool.cc:1074] tablet 11a8c270532947ba9b505aace44b56f6: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:25.081506 14224 rebalancer_tool.cc:1074] tablet c2e9f6eecbec45988b290775277dbddc: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:25.824963 14224 rebalancer_tool.cc:1136] tablet 11a8c270532947ba9b505aace44b56f6: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:26.425164 14224 rebalancer_tool.cc:1136] tablet 0e442bb664f44887aa8afdb62e1d97fc: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:26.470991 14224 rebalancer_tool.cc:1136] tablet a0106f7a40cb467eb4738890ac4e22d4: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
I20240503 14:00:26.492332 14224 rebalancer_tool.cc:1074] tablet e92d8f55591346158b9583feb958a8ca: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:26.505981 14224 rebalancer_tool.cc:1074] tablet 211de6f35a844e278e8c1eb1621e7c5f: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:26.517997 14224 rebalancer_tool.cc:1074] tablet 70ffc5888a5945d6a8e166420e6f1bd9: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:27.810230 14224 rebalancer_tool.cc:1136] tablet e92d8f55591346158b9583feb958a8ca: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:28.581534 14224 rebalancer_tool.cc:1136] tablet 5e6ca408c4254678b67a74f49a34881b: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:28.600826 14224 rebalancer_tool.cc:1074] tablet c42fee9fadbb4601a7fe32eb6a04e767: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:28.635576 14224 rebalancer_tool.cc:1136] tablet c42fee9fadbb4601a7fe32eb6a04e767: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:28.648553 14224 rebalancer_tool.cc:1136] tablet 70ffc5888a5945d6a8e166420e6f1bd9: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:29.100116 14224 rebalancer_tool.cc:1136] tablet c2e9f6eecbec45988b290775277dbddc: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:29.108901 14224 rebalancer_tool.cc:1136] tablet 211de6f35a844e278e8c1eb1621e7c5f: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
W20240503 14:00:29.726732 23059 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:29.371125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:46845 (local address 127.0.0.1:40532)
0503 14:00:29.441521 (+ 70396us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:29.441534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:29.441609 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:29.726139 (+284530us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":70314}
Illegal state: tablet server e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): unacceptable health status UNAVAILABLE
I20240503 14:00:29.914168 23012 tablet_copy_client.cc:308] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:00:29.914510 23104 raft_consensus.cc:348] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:29.914636 23104 raft_consensus.cc:729] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:29.914795 23104 consensus_queue.cc:260] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:29.915104 23104 ts_tablet_manager.cc:1398] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:29.916419 12319 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet c2e9f6eecbec45988b290775277dbddc from peer 84c63c6f551949f6a1d7c07760824361 ({username='slave'} at 127.0.99.132:33737)
I20240503 14:00:29.916509 12319 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet c2e9f6eecbec45988b290775277dbddc from peer 84c63c6f551949f6a1d7c07760824361 at {username='slave'} at 127.0.99.132:33737: session id = 84c63c6f551949f6a1d7c07760824361-c2e9f6eecbec45988b290775277dbddc
I20240503 14:00:29.917443 12319 tablet_copy_source_session.cc:205] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:29.917723 12318 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 84c63c6f551949f6a1d7c07760824361-a6a89230883347d4a64d96d321247012 received from {username='slave'} at 127.0.99.132:33737
I20240503 14:00:29.917770 12318 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 84c63c6f551949f6a1d7c07760824361-a6a89230883347d4a64d96d321247012 on tablet a6a89230883347d4a64d96d321247012 with peer 84c63c6f551949f6a1d7c07760824361
I20240503 14:00:29.924031 23012 ts_tablet_manager.cc:1875] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:00:29.926416 23012 ts_tablet_manager.cc:1888] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240503 14:00:29.926580 23012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2e9f6eecbec45988b290775277dbddc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:00:29.927366 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:29.931905 23012 tablet_copy_client.cc:783] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:29.932032 23012 tablet_copy_client.cc:647] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:29.958276 23012 tablet_copy_client.cc:523] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:29.959451 12429 raft_consensus.cc:1203] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:00:29.961814 23012 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Bootstrap starting.
I20240503 14:00:29.967545 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:29.982702 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.982800 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:29.993314 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:29.993669 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:29.996773 12298 raft_consensus.cc:1203] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 14:00:30.011399 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:30.012176 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:30.044428 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:30.051548 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.060642 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:30.064555 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.067744 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.081192 23012 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:00:30.081547 23012 tablet_bootstrap.cc:492] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Bootstrap complete.
I20240503 14:00:30.081714 23012 ts_tablet_manager.cc:1367] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Time spent bootstrapping tablet: real 0.120s user 0.026s sys 0.000s
I20240503 14:00:30.087626 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:30.098544 23012 raft_consensus.cc:348] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:30.099314 23012 raft_consensus.cc:729] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c63c6f551949f6a1d7c07760824361, State: Initialized, Role: FOLLOWER
I20240503 14:00:30.099448 23012 consensus_queue.cc:260] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:30.102242 23012 ts_tablet_manager.cc:1398] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240503 14:00:30.102705 12318 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 84c63c6f551949f6a1d7c07760824361-c2e9f6eecbec45988b290775277dbddc received from {username='slave'} at 127.0.99.132:33737
I20240503 14:00:30.102780 12318 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 84c63c6f551949f6a1d7c07760824361-c2e9f6eecbec45988b290775277dbddc on tablet c2e9f6eecbec45988b290775277dbddc with peer 84c63c6f551949f6a1d7c07760824361
W20240503 14:00:30.164767 12365 connection.cc:535] client connection to 127.0.99.136:34763 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:30.165313 12234 connection.cc:535] client connection to 127.0.99.136:34763 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:30.165678 12103 connection.cc:535] client connection to 127.0.99.136:34763 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:30.166189 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:30.166252 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:30.166289 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:30.166319 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:30.166354 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:30.167165 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:30.173522 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:30.164838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:34763 (local address 127.0.99.129:51935)
0503 14:00:30.164990 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:30.165005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:30.165088 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:30.172988 (+ 7900us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":82}
W20240503 14:00:30.174307 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:00:30.174551 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:30.165337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:34763 (local address 127.0.99.132:55761)
0503 14:00:30.165521 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:30.165534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:30.165600 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:30.174080 (+ 8480us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":118}
W20240503 14:00:30.174696 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:30.185794 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:30.186538 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.187444 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.188658 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:30.191782 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:30.205129 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:30.208506 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:30.222064 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:30.259680 12429 raft_consensus.cc:1203] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240503 14:00:30.303288 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:30.324497 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:30.349939 12015 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:30.350191 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.353070 12016 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:00:30.356447 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.357288 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:30.364029 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:30.372085 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:00:30.372160 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:30.380573 13754 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.381647 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.392119 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:30.393325 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:30.407284 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.407728 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.411290 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.417973 12429 raft_consensus.cc:1203] T c2e9f6eecbec45988b290775277dbddc P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240503 14:00:30.422031 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:30.438674 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:30.443094 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:30.450400 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:30.469359 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:30.490456 12037 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2089eeda1e0f4ff5926584d472e6631d"
dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55756
I20240503 14:00:30.490610 12037 raft_consensus.cc:595] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:30.501220 23285 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:00:30.501363 23285 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:30.501964 23285 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
W20240503 14:00:30.502969 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:30.503367 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:30.503482 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:00:30.503538 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:00:30.503640 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:30.503700 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:30.503837 23285 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:00:30.564469 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:30.583586 21626 raft_consensus.cc:979] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: : Instructing follower ca5386f84994496fa5376624391728ee to start an election
I20240503 14:00:30.583688 21626 raft_consensus.cc:1067] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Signalling peer ca5386f84994496fa5376624391728ee to start an election
I20240503 14:00:30.584311 12298 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2089eeda1e0f4ff5926584d472e6631d"
dest_uuid: "ca5386f84994496fa5376624391728ee"
from {username='slave'} at 127.0.99.129:38487
I20240503 14:00:30.584446 12298 raft_consensus.cc:482] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:30.584498 12298 raft_consensus.cc:3037] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:30.585680 12298 raft_consensus.cc:504] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:30.585994 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.586125 12298 leader_election.cc:288] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 2 election: Requested vote from peers 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489), b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075)
I20240503 14:00:30.586632 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2089eeda1e0f4ff5926584d472e6631d" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b67f337e10994234baedbf0b19c13513"
I20240503 14:00:30.586755 13906 raft_consensus.cc:3037] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:30.587508 12166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2089eeda1e0f4ff5926584d472e6631d" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "8b17d706d2344dbaae13cda9b0733143"
I20240503 14:00:30.587590 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2089eeda1e0f4ff5926584d472e6631d" candidate_uuid: "ca5386f84994496fa5376624391728ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "623e8dcb58544e558207d8e652dfed5f"
I20240503 14:00:30.587639 12166 raft_consensus.cc:3037] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:30.587718 12037 raft_consensus.cc:3032] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:30.587765 12037 raft_consensus.cc:729] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Running, Role: LEADER
I20240503 14:00:30.587843 13906 raft_consensus.cc:2454] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 2.
I20240503 14:00:30.587925 12037 consensus_queue.cc:260] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:30.588099 12037 raft_consensus.cc:3037] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:30.588632 12166 raft_consensus.cc:2454] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 2.
I20240503 14:00:30.589121 12037 raft_consensus.cc:2454] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5386f84994496fa5376624391728ee in term 2.
I20240503 14:00:30.592527 12235 leader_election.cc:302] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, b67f337e10994234baedbf0b19c13513, ca5386f84994496fa5376624391728ee; no voters:
I20240503 14:00:30.592952 21738 raft_consensus.cc:2781] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:30.593034 21738 raft_consensus.cc:686] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 2 LEADER]: Becoming Leader. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:30.593175 21738 consensus_queue.cc:237] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:30.594511 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:30.594702 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.595410 11891 catalog_manager.cc:5484] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee reported cstate change: term changed from 1 to 2, leader changed from 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) to ca5386f84994496fa5376624391728ee (127.0.99.131). New cstate: current_term: 2 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:30.605356 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:30.607096 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:30.624459 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.648059 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.651576 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.652103 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:30.665812 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:30.666321 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:30.670782 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.699962 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.706485 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:30.706732 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.712208 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.721431 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:30.739252 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240503 14:00:30.786787 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:30.844944 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.846562 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.848604 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:30.852772 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.853776 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:30.855686 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:30.921361 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:30.923439 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:30.929665 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:30.930351 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:30.931046 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.931267 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.948431 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:30.980943 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:30.981909 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:30.984030 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:30.986146 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:31.002705 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:31.003593 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:31.014376 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.014648 22932 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:00:31.014784 22932 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:31.015219 22932 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:00:31.017290 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:31.017470 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:31.017519 12103 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:31.017752 22932 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:00:31.055080 12167 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:31.055709 23635 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:31.072580 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:31.076124 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.076366 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.077489 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.080667 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.088457 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:31.131721 12037 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:31.132158 23635 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:31.135145 13906 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:00:31.136476 22931 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:00:31.169267 13775 consensus_queue.cc:237] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:31.171003 13906 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:31.171325 12037 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:31.171674 12298 raft_consensus.cc:1261] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:00:31.171975 22500 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:31.172329 22500 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:31.172612 22500 consensus_queue.cc:1035] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:00:31.176458 22500 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:31.179009 13906 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:31.179867 12037 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:31.182781 11892 catalog_manager.cc:5484] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: config changed from index 10 to 11, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:31.186988 12298 raft_consensus.cc:2932] T 5e6ca408c4254678b67a74f49a34881b P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:31.188248 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.188530 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:31.198892 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.204003 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.217096 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:31.234174 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:31.234441 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.245579 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.253338 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.257503 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:31.259833 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.279125 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.292119 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6ca408c4254678b67a74f49a34881b with delete_type TABLET_DATA_TOMBSTONED (TS 8b17d706d2344dbaae13cda9b0733143 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:33028
I20240503 14:00:31.292586 24017 tablet_replica.cc:330] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143: stopping tablet replica
I20240503 14:00:31.292798 24017 raft_consensus.cc:2229] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:31.293110 24017 raft_consensus.cc:2258] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:31.293741 24017 ts_tablet_manager.cc:1875] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:31.297456 24017 ts_tablet_manager.cc:1888] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 14:00:31.297538 24017 log.cc:1196] T 5e6ca408c4254678b67a74f49a34881b P 8b17d706d2344dbaae13cda9b0733143: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5e6ca408c4254678b67a74f49a34881b
I20240503 14:00:31.299837 11882 catalog_manager.cc:4830] TS 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): tablet 5e6ca408c4254678b67a74f49a34881b (table rebalance_test_table_0 [id=cf9cc41812f545f6bd3749ad1d6873a8]) successfully deleted
W20240503 14:00:31.329879 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:31.334944 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:31.338166 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.338559 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.339377 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.342178 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:31.353323 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:31.407938 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:31.432202 14221 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:00:14.041324 14222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:14.041539 14222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:14.046939 14222 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:00:15.050045 14222 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:00:15.604604 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.604755 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.604789 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.604856 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.604910 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.604985 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.605018 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.605048 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.605088 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.605127 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.605170 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.605201 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.605230 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.605260 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.605296 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:15.724758 14222 rebalancer_tool.cc:1074] tablet d1e9155230c74e92b6ee441de713a419: '84c63c6f551949f6a1d7c07760824361' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move scheduled
I20240503 14:00:15.862906 14222 rebalancer_tool.cc:1074] tablet c2e9f6eecbec45988b290775277dbddc: '8b17d706d2344dbaae13cda9b0733143' -> '1b4a61e8a11440e7a9f4b300d2442f75' move scheduled
I20240503 14:00:15.934165 14222 rebalancer_tool.cc:1074] tablet 2089eeda1e0f4ff5926584d472e6631d: 'ca5386f84994496fa5376624391728ee' -> 'b67f337e10994234baedbf0b19c13513' move scheduled
I20240503 14:00:16.037021 14222 rebalancer_tool.cc:1074] tablet 79b21b1e5292488aad442060b417f3c1: '623e8dcb58544e558207d8e652dfed5f' -> '66190085d1c04e068d5c0dec9c4bf07d' move scheduled
I20240503 14:00:16.229879 14222 rebalancer_tool.cc:1074] tablet a4b4e16598b740cb9d690c2442a3e994: '84c63c6f551949f6a1d7c07760824361' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move scheduled
I20240503 14:00:16.298175 14222 rebalancer_tool.cc:1074] tablet a6a89230883347d4a64d96d321247012: '8b17d706d2344dbaae13cda9b0733143' -> '1b4a61e8a11440e7a9f4b300d2442f75' move scheduled
I20240503 14:00:16.446132 14222 rebalancer_tool.cc:1074] tablet c42fee9fadbb4601a7fe32eb6a04e767: 'ca5386f84994496fa5376624391728ee' -> 'b67f337e10994234baedbf0b19c13513' move scheduled
I20240503 14:00:16.504751 14222 rebalancer_tool.cc:1084] tablet 5e6ca408c4254678b67a74f49a34881b: '623e8dcb58544e558207d8e652dfed5f' -> '66190085d1c04e068d5c0dec9c4bf07d' move ignored: Illegal state: tablet 5e6ca408c4254678b67a74f49a34881b: replica 66190085d1c04e068d5c0dec9c4bf07d is already present
I20240503 14:00:16.504853 14222 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:17.287819 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.288059 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.288208 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.288269 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.288349 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.288400 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.288442 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.288514 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.288561 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.288609 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.288655 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.288698 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.288744 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.288786 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:17.305550 14222 rebalancer_tool.cc:1074] tablet 92506cde7b084b14b88d4a915731eeb3: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:17.402803 14222 rebalancer_tool.cc:1074] tablet 4224ca4d93ce4784b52a7550468ce1b1: '8b17d706d2344dbaae13cda9b0733143' -> '66190085d1c04e068d5c0dec9c4bf07d' move scheduled
I20240503 14:00:17.429652 14222 rebalancer_tool.cc:1074] tablet f4f9cb87c70e43dbb0d850c275ebb8a5: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:17.586074 14222 rebalancer_tool.cc:1074] tablet b031f09b97664d1a94c14f0661181899: '623e8dcb58544e558207d8e652dfed5f' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move scheduled
I20240503 14:00:17.671659 14222 rebalancer_tool.cc:1084] tablet 0e442bb664f44887aa8afdb62e1d97fc: '84c63c6f551949f6a1d7c07760824361' -> 'b67f337e10994234baedbf0b19c13513' move ignored: Illegal state: tablet 0e442bb664f44887aa8afdb62e1d97fc: replica b67f337e10994234baedbf0b19c13513 is already present
I20240503 14:00:17.671761 14222 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:18.071362 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:18.071430 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:18.071475 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:18.071557 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:18.071619 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:18.071703 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:18.071760 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:18.071866 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:18.082528 14222 rebalancer_tool.cc:1074] tablet faa0a6ca00d442eaaf9fca7fc1259ec7: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:18.092768 14222 rebalancer_tool.cc:1084] tablet df13ec098dc548afac8c67195d25e7e8: '84c63c6f551949f6a1d7c07760824361' -> '1b4a61e8a11440e7a9f4b300d2442f75' move ignored: Illegal state: tablet df13ec098dc548afac8c67195d25e7e8: replica 1b4a61e8a11440e7a9f4b300d2442f75 is already present
I20240503 14:00:18.092855 14222 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:18.475433 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:18.475595 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:18.475670 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:18.475730 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:18.475824 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:18.475885 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:18.475919 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
I20240503 14:00:18.592464 14222 rebalancer_tool.cc:1074] tablet d837845c6f644bca83315c9f7c45246d: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:18.613564 14222 rebalancer_tool.cc:1074] tablet 5e6ca408c4254678b67a74f49a34881b: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:19.669943 14222 rebalancer_tool.cc:1136] tablet faa0a6ca00d442eaaf9fca7fc1259ec7: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:19.710192 14222 rebalancer_tool.cc:1136] tablet f4f9cb87c70e43dbb0d850c275ebb8a5: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:19.723985 14222 rebalancer_tool.cc:1136] tablet d837845c6f644bca83315c9f7c45246d: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:21.857699 14222 rebalancer_tool.cc:1136] tablet c2e9f6eecbec45988b290775277dbddc: '8b17d706d2344dbaae13cda9b0733143' -> '1b4a61e8a11440e7a9f4b300d2442f75' move completed: OK
I20240503 14:00:23.030364 14222 rebalancer_tool.cc:1136] tablet b031f09b97664d1a94c14f0661181899: '623e8dcb58544e558207d8e652dfed5f' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move completed: OK
I20240503 14:00:23.686365 14222 rebalancer_tool.cc:1123] tablet 79b21b1e5292488aad442060b417f3c1: '623e8dcb58544e558207d8e652dfed5f' -> '66190085d1c04e068d5c0dec9c4bf07d' move is abandoned: Not found: Server with UUID 623e8dcb58544e558207d8e652dfed5f not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:23.713588 14222 rebalancer_tool.cc:1136] tablet a6a89230883347d4a64d96d321247012: '8b17d706d2344dbaae13cda9b0733143' -> '1b4a61e8a11440e7a9f4b300d2442f75' move completed: OK
I20240503 14:00:23.722829 14222 rebalancer_tool.cc:1074] tablet dcdc3ae5232a49cdb003d9a2e3cf6d92: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:23.731160 14222 rebalancer_tool.cc:1074] tablet a0106f7a40cb467eb4738890ac4e22d4: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:23.767452 14222 rebalancer_tool.cc:1074] tablet 4a608f683c4f41f3bbdfc9b4c219ce49: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:23.821517 14222 rebalancer_tool.cc:1136] tablet 4a608f683c4f41f3bbdfc9b4c219ce49: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:23.905660 14222 rebalancer_tool.cc:1136] tablet dcdc3ae5232a49cdb003d9a2e3cf6d92: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
I20240503 14:00:24.022853 14222 rebalancer_tool.cc:1136] tablet 5e6ca408c4254678b67a74f49a34881b: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:24.077512 14222 rebalancer_tool.cc:1136] tablet c42fee9fadbb4601a7fe32eb6a04e767: 'ca5386f84994496fa5376624391728ee' -> 'b67f337e10994234baedbf0b19c13513' move completed: OK
I20240503 14:00:24.089401 14222 rebalancer_tool.cc:1136] tablet a0106f7a40cb467eb4738890ac4e22d4: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:24.139498 14222 rebalancer_tool.cc:1136] tablet 4224ca4d93ce4784b52a7550468ce1b1: '8b17d706d2344dbaae13cda9b0733143' -> '66190085d1c04e068d5c0dec9c4bf07d' move completed: OK
I20240503 14:00:24.764076 14222 rebalancer_tool.cc:1136] tablet 2089eeda1e0f4ff5926584d472e6631d: 'ca5386f84994496fa5376624391728ee' -> 'b67f337e10994234baedbf0b19c13513' move completed: OK
I20240503 14:00:24.795434 14222 rebalancer_tool.cc:1136] tablet d1e9155230c74e92b6ee441de713a419: '84c63c6f551949f6a1d7c07760824361' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move completed: OK
I20240503 14:00:24.815819 14222 rebalancer_tool.cc:1136] tablet a4b4e16598b740cb9d690c2442a3e994: '84c63c6f551949f6a1d7c07760824361' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move completed: OK
I20240503 14:00:24.957423 14222 rebalancer_tool.cc:1074] tablet 211de6f35a844e278e8c1eb1621e7c5f: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:24.965306 14222 rebalancer_tool.cc:1074] tablet e92d8f55591346158b9583feb958a8ca: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:25.020648 14222 rebalancer_tool.cc:1074] tablet 3481715d755a4277b0905b0832fad4af: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:25.030115 14222 rebalancer_tool.cc:1074] tablet 6c4f3bc64e7145e2b692beab730d1496: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:26.460341 14222 rebalancer_tool.cc:1136] tablet 211de6f35a844e278e8c1eb1621e7c5f: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:27.319548 14222 rebalancer_tool.cc:1136] tablet 92506cde7b084b14b88d4a915731eeb3: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:27.336385 14222 rebalancer_tool.cc:1074] tablet 11a8c270532947ba9b505aace44b56f6: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:27.403477 14222 rebalancer_tool.cc:1074] tablet 70ffc5888a5945d6a8e166420e6f1bd9: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:27.423559 14222 rebalancer_tool.cc:1136] tablet 70ffc5888a5945d6a8e166420e6f1bd9: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:27.433329 14222 rebalancer_tool.cc:1136] tablet 11a8c270532947ba9b505aace44b56f6: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:27.446131 14222 rebalancer_tool.cc:1136] tablet 6c4f3bc64e7145e2b692beab730d1496: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:27.495882 14222 rebalancer_tool.cc:1136] tablet e92d8f55591346158b9583feb958a8ca: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:27.982663 14222 rebalancer_tool.cc:1136] tablet 3481715d755a4277b0905b0832fad4af: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
W20240503 14:00:28.532459 14222 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server e85b0283a0fd4e3ba81b4170f3cdacb5 to server 84c63c6f551949f6a1d7c07760824361
I20240503 14:00:28.608219 14222 rebalancer_tool.cc:1074] tablet 5e6ca408c4254678b67a74f49a34881b: '8b17d706d2344dbaae13cda9b0733143' -> 'ca5386f84994496fa5376624391728ee' move scheduled
I20240503 14:00:28.741531 14222 rebalancer_tool.cc:1074] tablet 79b21b1e5292488aad442060b417f3c1: 'e85b0283a0fd4e3ba81b4170f3cdacb5' -> '1b4a61e8a11440e7a9f4b300d2442f75' move scheduled
I20240503 14:00:28.891779 14222 rebalancer_tool.cc:1074] tablet faa0a6ca00d442eaaf9fca7fc1259ec7: 'b67f337e10994234baedbf0b19c13513' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move scheduled
I20240503 14:00:29.001317 14222 rebalancer_tool.cc:1074] tablet f5c06806a19e4f4592d26ed4acc651cd: '1b4a61e8a11440e7a9f4b300d2442f75' -> '623e8dcb58544e558207d8e652dfed5f' move scheduled
I20240503 14:00:29.027009 14222 rebalancer_tool.cc:1074] tablet 4224ca4d93ce4784b52a7550468ce1b1: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:29.161939 14222 rebalancer_tool.cc:1074] tablet a6a89230883347d4a64d96d321247012: '1b4a61e8a11440e7a9f4b300d2442f75' -> '84c63c6f551949f6a1d7c07760824361' move scheduled
W20240503 14:00:29.728924 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:29.592467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:46845 (local address 127.0.0.1:42492)
0503 14:00:29.605347 (+ 12880us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:29.605362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:29.606038 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:29.727116 (+121078us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":12819}
W20240503 14:00:30.172560 23316 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:29.972689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:34763 (local address 127.0.0.1:53234)
0503 14:00:30.047391 (+ 74702us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:30.047403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:30.047464 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:30.172456 (+124992us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":74630}
I20240503 14:00:30.203447 14222 rebalancer_tool.cc:1123] tablet f5c06806a19e4f4592d26ed4acc651cd: '1b4a61e8a11440e7a9f4b300d2442f75' -> '623e8dcb58544e558207d8e652dfed5f' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:31.180197 14222 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): unacceptable health status UNAVAILABLE
I20240503 14:00:31.452198 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:31.454440 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.458528 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.459017 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.460610 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.476121 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:31.480793 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:31.483521 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.484266 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:31.495640 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:31.496466 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.505568 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:31.537107 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.548676 12298 consensus_queue.cc:237] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:00:31.549835 13906 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:31.550051 12166 raft_consensus.cc:1261] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:00:31.550261 23913 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:31.550390 22931 consensus_queue.cc:1035] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:00:31.552016 22931 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:31.553411 13906 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:31.554797 12166 raft_consensus.cc:2932] T 2089eeda1e0f4ff5926584d472e6631d P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:31.557171 11891 catalog_manager.cc:5484] T 2089eeda1e0f4ff5926584d472e6631d P b67f337e10994234baedbf0b19c13513 reported cstate change: config changed from index 6 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:00:31.560307 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet 2089eeda1e0f4ff5926584d472e6631d with delete_type TABLET_DATA_TOMBSTONED (TS 623e8dcb58544e558207d8e652dfed5f not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:60852
I20240503 14:00:31.560690 24139 tablet_replica.cc:330] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: stopping tablet replica
I20240503 14:00:31.560868 24139 raft_consensus.cc:2229] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:31.561035 24139 raft_consensus.cc:2258] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:31.561939 24139 ts_tablet_manager.cc:1875] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:31.564494 24139 ts_tablet_manager.cc:1888] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 14:00:31.564575 24139 log.cc:1196] T 2089eeda1e0f4ff5926584d472e6631d P 623e8dcb58544e558207d8e652dfed5f: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2089eeda1e0f4ff5926584d472e6631d
I20240503 14:00:31.565100 12298 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92506cde7b084b14b88d4a915731eeb3"
dest_uuid: "ca5386f84994496fa5376624391728ee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46158
I20240503 14:00:31.565171 11882 catalog_manager.cc:4830] TS 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): tablet 2089eeda1e0f4ff5926584d472e6631d (table rebalance_test_table_7 [id=fbf10f839f48498b898afc6af4a813a6]) successfully deleted
I20240503 14:00:31.565223 12298 raft_consensus.cc:595] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Received request to transfer leadership
I20240503 14:00:31.635366 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.640028 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.643378 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.644105 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.646494 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:31.651741 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.655148 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:31.680025 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.687517 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.696666 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.698832 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:31.699407 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240503 14:00:31.765744 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:31.802716 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.803516 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:31.804239 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.817698 12277 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:00:31.818918 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.822330 13755 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:31.825147 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:31.830160 22931 raft_consensus.cc:979] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: : Instructing follower 66190085d1c04e068d5c0dec9c4bf07d to start an election
I20240503 14:00:31.830265 22931 raft_consensus.cc:1067] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Signalling peer 66190085d1c04e068d5c0dec9c4bf07d to start an election
I20240503 14:00:31.830647 13775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92506cde7b084b14b88d4a915731eeb3"
dest_uuid: "66190085d1c04e068d5c0dec9c4bf07d"
from {username='slave'} at 127.0.99.131:46779
I20240503 14:00:31.830776 13775 raft_consensus.cc:482] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:00:31.830834 13775 raft_consensus.cc:3037] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:31.831921 13775 raft_consensus.cc:504] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:31.832351 13775 leader_election.cc:288] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [CANDIDATE]: Term 2 election: Requested vote from peers ca5386f84994496fa5376624391728ee (127.0.99.131:36399), b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075), 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:00:31.832702 12429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92506cde7b084b14b88d4a915731eeb3" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "84c63c6f551949f6a1d7c07760824361"
I20240503 14:00:31.832828 12429 raft_consensus.cc:3037] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:31.832883 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92506cde7b084b14b88d4a915731eeb3" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b67f337e10994234baedbf0b19c13513"
I20240503 14:00:31.832999 13906 raft_consensus.cc:3037] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:31.833811 12429 raft_consensus.cc:2454] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:31.833971 13906 raft_consensus.cc:2454] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:31.834556 13710 leader_election.cc:302] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 66190085d1c04e068d5c0dec9c4bf07d, 84c63c6f551949f6a1d7c07760824361, b67f337e10994234baedbf0b19c13513; no voters:
I20240503 14:00:31.834739 23969 raft_consensus.cc:2781] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:00:31.834908 23969 raft_consensus.cc:686] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Becoming Leader. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Running, Role: LEADER
I20240503 14:00:31.835043 23969 consensus_queue.cc:237] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:31.836673 12298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92506cde7b084b14b88d4a915731eeb3" candidate_uuid: "66190085d1c04e068d5c0dec9c4bf07d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ca5386f84994496fa5376624391728ee"
I20240503 14:00:31.836812 12298 raft_consensus.cc:3032] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:00:31.836882 12298 raft_consensus.cc:729] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Running, Role: LEADER
I20240503 14:00:31.837009 12298 consensus_queue.cc:260] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:31.837221 12298 raft_consensus.cc:3037] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:00:31.837445 11888 catalog_manager.cc:5484] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: term changed from 1 to 2, leader changed from ca5386f84994496fa5376624391728ee (127.0.99.131) to 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133). New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:31.838146 12298 raft_consensus.cc:2454] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66190085d1c04e068d5c0dec9c4bf07d in term 2.
I20240503 14:00:31.856245 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:31.856371 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:31.859364 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.868634 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:31.869863 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:31.870684 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.875998 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:31.900449 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:31.980696 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:31.982875 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:31.987417 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:31.991437 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:31.996013 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:31.998636 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.018939 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:32.052397 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:32.056473 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:32.058956 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.059738 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:32.062695 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:32.065976 23553 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:00:32.066062 23553 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:32.066437 23553 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
W20240503 14:00:32.068065 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:32.068393 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:32.069319 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:32.069412 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:00:32.069463 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:00:32.070253 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:32.070315 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:32.070574 23553 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:00:32.073431 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:32.078940 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:32.106526 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.137699 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:32.159577 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:32.161092 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:32.163622 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.164841 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:32.171555 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:32.174516 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:32.175652 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:32.201439 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:32.224133 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.225934 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:32.226917 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.232128 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:32.232617 12429 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:32.232995 24308 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:32.237112 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:32.238761 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:32.300547 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.302529 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:32.320737 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.322853 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:32.323331 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.324534 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:32.329097 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:32.341197 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:32.351878 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.351929 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:32.368858 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:32.374317 12298 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:32.374792 24308 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:00:32.378233 13906 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:00:32.378683 24540 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 14:00:32.417809 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:32.449566 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:32.452822 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:32.454221 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:32.455807 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.458643 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:32.465070 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:32.479022 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.491930 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.499589 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:32.502265 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.506017 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:32.514631 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:32.518395 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:32.524410 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:32.524976 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.525429 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:32.530746 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.531976 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.604570 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:32.631362 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:32.635989 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:32.643025 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.646145 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:32.648175 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:32.653004 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.654110 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:32.665022 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:32.671999 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.685820 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:32.688995 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:32.710712 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:32.723392 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.724566 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.734390 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.738538 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.745069 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:32.751001 13757 consensus_queue.cc:237] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:00:32.752156 12429 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:32.752282 13906 raft_consensus.cc:1261] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Refusing update from remote peer 66190085d1c04e068d5c0dec9c4bf07d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:00:32.752568 24540 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:32.752712 24542 consensus_queue.cc:1035] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:00:32.754223 24308 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:32.754621 13906 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:32.754823 12429 raft_consensus.cc:2932] T 92506cde7b084b14b88d4a915731eeb3 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:32.756986 11892 catalog_manager.cc:5484] T 92506cde7b084b14b88d4a915731eeb3 P b67f337e10994234baedbf0b19c13513 reported cstate change: config changed from index 8 to 10, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "66190085d1c04e068d5c0dec9c4bf07d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:00:32.759752 12277 tablet_service.cc:1514] Processing DeleteTablet for tablet 92506cde7b084b14b88d4a915731eeb3 with delete_type TABLET_DATA_TOMBSTONED (TS ca5386f84994496fa5376624391728ee not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51046
I20240503 14:00:32.760058 24749 tablet_replica.cc:330] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: stopping tablet replica
I20240503 14:00:32.760175 24749 raft_consensus.cc:2229] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:32.760275 24749 raft_consensus.cc:2258] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:32.760939 24749 ts_tablet_manager.cc:1875] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:00:32.761379 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:00:32.763078 24749 ts_tablet_manager.cc:1888] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:00:32.763149 24749 log.cc:1196] T 92506cde7b084b14b88d4a915731eeb3 P ca5386f84994496fa5376624391728ee: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/92506cde7b084b14b88d4a915731eeb3
I20240503 14:00:32.763665 11882 catalog_manager.cc:4830] TS ca5386f84994496fa5376624391728ee (127.0.99.131:36399): tablet 92506cde7b084b14b88d4a915731eeb3 (table rebalance_test_table_9 [id=40371d9909ff4960ac96d09aac7936cc]) successfully deleted
W20240503 14:00:32.840979 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.921380 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:00:32.966539 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:32.976579 22932 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:00:32.976683 22932 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:32.977013 22932 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:00:32.979460 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:32.979848 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:32.979928 12103 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:32.980180 22932 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:00:32.983209 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:32.988777 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.991070 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:32.991150 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:32.992527 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:33.003983 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.014264 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:33.064203 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:33.065409 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.075474 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:33.081259 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.092346 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.105528 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:33.154062 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:33.154326 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.158427 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.164084 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:33.170202 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:33.170246 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.234006 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:33.235849 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.237484 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.240337 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:33.244875 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.250933 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:33.334141 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:33.339195 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:33.342553 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:33.345101 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:33.356235 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.357600 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.361027 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:33.470726 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:33.471381 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:33.473062 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:33.479158 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:33.486172 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.487326 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.489898 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.499415 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:33.501598 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.507571 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:33.513897 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.523378 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:33.533430 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:33.537536 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:33.640934 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:33.645761 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:33.646252 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:33.651005 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.662235 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.671216 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.673064 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.680910 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:33.688431 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:33.691797 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:33.695027 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.697887 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.794680 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:33.796387 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.801120 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.805356 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:33.811408 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:33.813649 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.911276 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:33.914702 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:33.915864 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:33.918000 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:33.919864 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:33.944612 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:33.960301 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:34.027782 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:34.033586 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.036680 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:34.037201 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:34.039196 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:34.054297 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.073005 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:34.086011 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:34.092758 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.093135 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:34.095857 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:34.098780 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:34.103919 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:34.138891 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:34.198150 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.203965 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:34.207065 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.207669 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:34.209128 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:34.224164 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:34.240419 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:34.251901 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:34.265336 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.267532 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.270880 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:34.276379 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:34.303691 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:34.321352 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:34.365759 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:34.377684 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:34.377787 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:34.378365 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.379508 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:34.379659 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:34.381017 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.404569 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:34.426265 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:34.432921 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:34.529426 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:34.536037 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.536852 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.537325 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:34.539366 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:34.539620 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:34.546408 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:34.549312 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:34.631429 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:34.631608 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:34.632302 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:34.633220 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.634289 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:34.636690 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:34.649717 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:34.707367 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:34.723861 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:34.725493 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.727217 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.728888 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:34.733731 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240503 14:00:34.737797 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:34.743549 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:34.748632 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:34.779107 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:00:34.792191 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20240503 14:00:34.792841 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:34.807188 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:34.813730 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:34.824446 23553 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:00:34.824548 23553 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:34.825024 23553 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:34.825922 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:34.826053 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:00:34.826136 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:00:34.826512 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:34.827044 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:34.827113 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:34.827301 23553 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:00:34.846060 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:34.846930 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:34.849916 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.851068 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.861315 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:34.861359 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:34.868422 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:34.879206 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:34.912530 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 14:00:34.917349 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:34.953326 22932 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:00:34.953454 22932 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:34.953881 22932 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:00:34.956028 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:34.956171 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:34.956234 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:34.956707 22932 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:00:34.973588 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:34.973793 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.976703 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:34.988404 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:34.995141 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:35.000489 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:35.000751 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:35.014209 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.018079 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:35.021615 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:35.025313 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:35.032946 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:35.035988 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:35.051437 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:35.051483 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:35.067587 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:35.094296 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:35.122332 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.126631 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:35.126791 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:35.135974 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:35.144495 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:35.144536 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.172205 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:35.174677 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.185945 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.186390 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:35.195840 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:35.200372 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:35.201508 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:35.216238 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:35.284437 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:35.302003 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:35.302990 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:35.306391 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.309924 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:35.316505 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:35.326958 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:35.347389 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:35.408859 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:35.417234 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:35.430078 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:00:35.452733 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:35.463618 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:35.496995 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.497813 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:35.499580 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:35.508396 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.509567 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:35.516543 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:35.521356 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:35.612056 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:35.612740 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.619184 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:35.621450 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.622473 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:35.623632 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:35.624122 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:35.669224 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:35.671509 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.673240 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:35.681668 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.682173 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:35.700727 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:35.722719 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:00:35.784329 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.784559 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:35.785396 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.789077 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:35.789781 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:35.817673 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:35.838038 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:35.896185 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:35.896814 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:35.897990 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:35.908790 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.911799 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:35.921334 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:35.996474 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:35.997918 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:35.999531 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.006230 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.012681 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:36.017923 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:36.018522 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:36.057300 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:36.057646 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:36.060318 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.072477 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:36.078995 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.079604 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:36.085121 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:36.143888 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:36.148989 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:36.149734 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.156702 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:36.156981 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:36.158512 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.217535 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:36.220811 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:36.222621 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:36.222728 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.231660 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.239413 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:36.388484 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:36.407864 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:36.456686 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:36.457777 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:36.461346 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:36.462801 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.472647 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:36.477743 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:36.539646 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:36.558532 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:36.562573 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.563532 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.565282 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:36.568532 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:36.594837 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:36.595854 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:36.630185 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:36.633427 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:36.637315 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.644775 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:36.648057 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:36.649938 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.725523 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.726063 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:36.726982 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:36.728123 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:36.737295 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:36.741585 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:36.742249 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:36.775629 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:36.838698 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:36.842041 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:36.844058 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:36.845273 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.846649 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:36.853914 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:36.868455 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:36.890400 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:36.925711 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:36.971763 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.972405 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:36.974164 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:36.978894 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:36.981420 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:36.984203 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:36.998868 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:37.000401 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:37.017766 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:37.018450 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.026202 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:37.028736 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:37.040156 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:37.086886 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:37.109732 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:37.132623 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:37.138551 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.142877 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:37.150070 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:37.158535 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:37.170640 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:37.170989 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:37.171353 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:37.173327 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.176899 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:37.178355 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:37.190052 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:37.197170 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:37.216866 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:37.242342 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:37.289805 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:37.309873 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:37.310124 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:37.314325 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:37.323036 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:37.514256 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:37.515862 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:37.527745 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.529711 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.530864 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.534510 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:37.541539 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:37.543695 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.544579 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:37.544883 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:37.549705 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:37.551414 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:37.563014 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:37.584059 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:37.620731 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:37.683893 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:37.687541 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:37.699782 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.701159 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:37.703491 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:37.704108 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.704205 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:37.704675 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:37.705457 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:37.707170 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:37.712656 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:37.746570 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:37.746623 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:37.746682 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:37.749255 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:37.758909 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:37.784611 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:37.834667 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:37.836625 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:37.838765 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:37.841849 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:37.843106 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:37.847632 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:37.849716 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.858173 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:37.891880 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:37.910900 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:37.942759 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:37.946823 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:37.957247 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.958395 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:37.960143 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:37.968369 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:37.990046 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:00:37.998883 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:38.065853 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:38.068897 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:38.072911 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:38.075943 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.086134 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.088080 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:38.091459 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:38.095607 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:38.101217 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.111663 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:38.118837 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:38.126751 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:38.133384 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.214166 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:38.215042 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:00:38.215615 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:38.216204 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:38.219508 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:38.219782 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.231640 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:38.263599 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:38.313546 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:38.387073 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:38.421479 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.423009 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:38.427466 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:38.432307 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:38.436161 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:38.444577 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.533593 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:38.542611 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.543906 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:38.555461 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:38.556592 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:38.558029 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.571939 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:38.576115 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.579310 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:38.579701 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.602152 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:38.608678 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:38.698725 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:38.701431 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:38.701601 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:38.705927 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:38.715226 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:38.716398 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:38.716739 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:38.717754 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.734959 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.739871 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:38.742036 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:38.747567 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:38.778529 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:38.822863 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:38.834100 23553 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:00:38.834223 23553 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:38.834666 23553 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:38.835086 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:38.835197 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:00:38.835278 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:00:38.835880 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:38.836315 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:38.836390 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:38.836525 23553 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:00:38.854890 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.856462 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:38.857018 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.858901 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:38.870669 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:38.879731 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:38.901120 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:38.972594 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:38.973204 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.974659 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:38.985776 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:38.986065 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:39.004997 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:39.076086 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.076134 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:39.084399 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:39.085048 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.085276 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:39.104480 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:39.105165 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:39.186712 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:39.188870 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:39.191035 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.193488 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:39.199579 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.216634 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:39.238088 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:39.289753 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:39.292269 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:39.293347 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.295136 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:39.303392 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:39.305146 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:39.306520 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.319000 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:39.345671 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:39.405156 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.407034 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:39.412377 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:39.417068 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:39.417477 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.423410 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:39.449751 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:39.470010 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:39.482384 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:39.482590 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:39.517251 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.517948 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:39.518723 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:39.520723 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:39.521370 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:39.545161 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:39.546340 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:39.559224 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:39.563787 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:39.565766 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:39.569101 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.572082 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.578550 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:39.590090 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:39.593375 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:39.665361 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:39.696003 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.699553 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:39.701644 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:39.706012 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:39.706527 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.710525 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.717455 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.719728 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:39.731042 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:39.732376 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:39.753989 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:39.754328 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:39.761795 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:39.831811 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 14:00:39.845103 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240503 14:00:39.845706 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:39.849504 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:39.849928 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:39.850767 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:39.860347 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:39.861241 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.861954 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:39.877755 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:39.910125 22932 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:00:39.910229 22932 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:39.910584 22932 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:00:39.911445 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:39.911531 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:39.911595 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:39.911741 22932 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:00:39.948545 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20240503 14:00:39.949306 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:39.976135 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:39.980747 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:40.036746 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:40.055936 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:40.060873 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.061646 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:40.062168 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:40.062426 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.063350 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:40.071058 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:40.081212 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 35 similar messages]
W20240503 14:00:40.114782 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:40.169009 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:40.172485 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.176421 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.177449 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:40.180258 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:40.185360 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:40.200356 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:40.202837 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:40.205132 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:40.218211 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.218950 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:40.227404 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.233306 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:40.237927 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:40.251919 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:40.321959 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:40.322077 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:40.324937 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:40.324996 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.326912 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:40.328325 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:40.336591 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:40.384124 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:40.430133 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:40.438997 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:40.439049 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:40.448482 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.449784 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:40.451691 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:40.451980 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:40.452072 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:40.465965 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:40.472967 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:40.486683 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:40.559715 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:40.560446 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.564224 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:40.575979 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:40.588455 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:40.597044 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:40.599213 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:40.602262 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.605587 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:40.607069 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:40.608510 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:40.618865 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.628397 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:40.677721 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:00:40.741063 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:40.744807 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:40.751271 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.751405 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:40.759932 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:40.760367 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:40.778056 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:40.788487 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.788789 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.795854 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:40.806313 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:40.817414 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:40.887940 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:00:40.918794 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:40.968641 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:41.108282 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:41.113512 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:41.116845 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:41.129791 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.136533 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:41.137809 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:41.140353 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:41.154440 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:41.155704 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.165695 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.165706 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.170526 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:41.193272 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:41.292115 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.295611 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:41.295823 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.310093 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:41.310808 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:41.311398 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:41.327152 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:41.328025 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:41.328639 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:41.334594 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:41.343400 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:41.345520 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.370728 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:41.406096 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:41.450281 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:41.451520 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:41.452477 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.454793 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:41.458963 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.462664 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:41.489225 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:41.551334 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:41.552047 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.553684 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:41.556751 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.559814 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:41.576203 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:41.613760 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:41.675721 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:41.683072 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:41.683315 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.683351 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:41.685710 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.696184 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:41.701428 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:41.710144 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:41.713994 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:41.722623 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:41.723819 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.724817 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:41.750643 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:41.760644 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:41.814754 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:41.835187 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:41.836835 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:41.838378 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:41.838603 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.845678 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:41.853519 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:41.865708 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:41.870059 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:41.890537 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:41.914404 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:41.921651 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:42.012176 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:42.043462 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:42.044735 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.046435 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:42.055073 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:42.057622 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.068245 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:42.084404 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:42.142303 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:42.146587 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:42.152410 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:42.158059 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:42.160861 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.163148 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:42.165287 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.178382 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:42.203394 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:42.204936 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.213033 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:42.216135 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.221210 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:42.228451 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:42.244428 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:42.248090 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:42.297477 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:42.324931 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:42.329000 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.331683 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:42.341501 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:42.350204 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.358562 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:42.384006 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.387575 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:42.388597 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:42.388665 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:42.390913 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.402527 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:42.404050 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:42.441612 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:42.505018 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.505115 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:42.507544 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:42.509969 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:42.519224 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:42.531517 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:42.540784 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:42.619630 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:42.620190 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:42.620954 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.628777 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:42.630481 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.631028 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:42.675709 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:42.676096 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:42.684846 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:42.705665 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:42.709870 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:42.710295 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:42.710582 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.720537 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.721503 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:42.722582 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:42.726675 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:42.733182 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:42.794255 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:42.798946 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:42.815820 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:42.815907 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:42.821069 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:42.828511 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:42.831105 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.839419 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:42.855954 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:00:42.856768 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:42.934453 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:42.935503 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:42.935709 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.943328 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:42.948571 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:42.967411 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:42.977670 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:43.051055 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:43.051930 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:43.053953 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:43.054222 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:43.060046 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:43.063098 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:43.073367 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:43.153447 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:43.161809 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:43.166519 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:43.167934 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.168220 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.168593 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:43.189435 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:43.194584 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:43.199512 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:43.199716 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:43.202365 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.215291 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:43.230782 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:43.234016 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:43.236951 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:43.244943 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:43.317512 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:43.326838 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:43.332077 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:43.340997 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.352622 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.362382 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:43.386291 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:43.389549 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:43.397184 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.397727 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.398778 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:43.399909 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:43.411566 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:00:43.502379 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.502699 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:43.504376 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.505827 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:43.507354 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:43.509153 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:43.671003 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.671116 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:43.679353 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:43.681301 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:43.687369 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:43.687914 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:43.691843 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:43.758836 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:43.792208 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:43.799289 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.804319 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:43.805608 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:43.812582 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.815709 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:43.825332 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:43.854010 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:43.856568 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:43.857132 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.866823 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.875399 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:43.875404 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:43.962527 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:43.972923 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.973398 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:43.982329 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:43.983390 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:43.984231 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:44.099414 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:44.101640 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:44.107836 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:44.110005 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:44.110816 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:44.125025 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:44.126569 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:44.141427 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:44.178402 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:44.182716 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:44.185281 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:44.187350 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:44.188292 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:44.196453 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:44.199460 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:44.199726 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:44.215840 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:44.262583 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:44.262923 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:44.268357 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:44.273285 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:44.273553 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:44.289098 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:44.291637 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:44.292119 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:44.300807 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:44.328325 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:44.329959 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:44.335294 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:44.336827 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:44.339160 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:44.350234 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:44.364032 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:44.380185 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:44.423173 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:44.443372 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:44.445042 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:44.446002 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:44.449940 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:44.453061 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:44.466995 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:44.472559 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:44.556685 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:44.614120 27515 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:00:44.614253 27515 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:44.614706 27515 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:44.615271 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:44.615412 12036 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:00:44.615482 12036 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:00:44.619781 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:44.620170 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:44.620246 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:44.620375 27515 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:00:44.662374 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:44.666529 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:44.668090 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:44.668123 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:44.676743 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:44.677462 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:44.697898 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:44.776368 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:44.791563 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:44.792402 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:44.793550 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:44.810792 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:44.812677 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:44.822594 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:44.829377 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:44.831936 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:44.832998 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:44.844144 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:00:44.845166 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:44.845818 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:44.846602 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:44.853988 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:44.857973 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 163 similar messages]
W20240503 14:00:44.859004 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:44.920432 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:44.950894 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:44.956324 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:44.958094 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:44.960026 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:44.968459 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:44.986354 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:45.045626 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 44 similar messages]
I20240503 14:00:45.068686 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:45.069909 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:45.072526 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:45.074136 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:45.085646 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:45.088095 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:45.111306 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:45.133889 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:45.140877 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:45.146965 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:45.168671 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 14:00:45.196260 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:45.205247 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:45.205492 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:45.208508 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:45.210458 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:45.223155 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:45.223374 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:45.226135 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:45.227885 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:45.241380 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:45.247498 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:45.258849 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:45.264771 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:45.266597 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:45.271906 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:45.279179 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:45.280193 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:45.282296 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:45.285881 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:45.358314 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:45.367185 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:45.372063 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:45.383168 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:45.387344 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:45.393170 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:45.394343 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:45.414664 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:45.420221 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:45.422641 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:45.423792 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:45.425748 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:45.432137 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:45.553601 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:45.667155 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:45.671399 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:45.744921 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:45.750979 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:45.766175 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:45.768841 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:45.792105 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:45.792086 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:45.812157 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:45.816805 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:45.821808 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:45.829016 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:45.847589 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:45.852105 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:45.852566 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:45.898504 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:45.922842 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:00:45.996711 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:46.004484 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:46.006719 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.009366 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.019968 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:46.021158 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:46.023527 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:46.037897 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:46.038095 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:46.060568 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:46.061758 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:46.081787 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:46.092250 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.164290 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:46.166800 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.166963 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.169869 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:46.172238 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:46.186344 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:46.211356 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:00:46.258371 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:46.279628 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:46.280906 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:46.281049 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.282397 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:46.288946 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:46.294847 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.388983 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:46.390873 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:46.396273 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:46.403636 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:46.404361 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.417582 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.455689 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:46.456071 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.459619 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.464565 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:46.471208 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:46.478094 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:46.503917 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:46.552134 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:46.554101 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.555302 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:46.561617 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:46.562412 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.565351 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:46.578899 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:46.593835 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:46.604326 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:46.637811 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:46.700109 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:46.704267 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:46.778075 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:46.779701 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:46.782593 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.786383 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:46.795715 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.811097 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:46.889314 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:46.893071 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:46.895623 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:46.903707 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:46.923359 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:46.923360 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.926817 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:46.938407 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:46.938822 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.940495 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:46.943455 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:46.966774 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:46.967329 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:46.984629 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:47.006845 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:47.011283 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:47.040127 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:47.078780 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:47.081358 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:47.082863 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.083416 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:47.084297 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.088210 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:47.095470 29295 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:00:47.095592 29295 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:47.095988 29295 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:00:47.100991 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:47.102021 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:47.102093 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:47.102293 29295 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:00:47.105119 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:47.110483 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:47.115890 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:47.128631 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:47.130926 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:47.142906 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.144387 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:47.158142 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:47.163766 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:47.246354 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.246407 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:47.251121 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:47.255762 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:47.261732 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:47.273591 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:47.321514 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:47.357950 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:47.362064 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:47.364650 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:47.370908 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:47.375128 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.391788 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:47.394923 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:47.455988 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:47.465135 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.465854 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:47.469534 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:47.473590 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:47.475077 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.478139 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:47.541756 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:47.547288 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:47.560168 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:47.567116 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:47.582456 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:47.584553 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.588205 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.590015 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:47.599311 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:47.607981 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:47.675444 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:47.691802 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:47.693230 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:47.694761 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:47.706588 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.706900 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:47.712818 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:47.716723 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:47.723073 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:47.772405 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:47.772929 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:47.807592 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.808027 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:47.811035 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:47.811686 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:47.812304 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.813242 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:47.928305 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:47.931063 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:47.932207 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:47.947686 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:47.949944 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:47.950253 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.952646 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:47.953119 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:00:47.959192 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:47.972048 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:47.972047 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:47.974215 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.977833 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:47.997891 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:47.999281 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.006800 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:48.006979 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:48.096354 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:48.096410 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:48.096693 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.114552 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:48.120342 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.129225 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:48.170231 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:48.172199 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:48.172397 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:48.180490 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.181679 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:48.187393 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:48.188251 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:48.214367 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:48.267295 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:48.276836 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:48.276978 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:48.282394 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:48.285976 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.289021 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:48.289175 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:48.293455 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:48.346098 29750 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:00:48.346262 29750 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:48.346848 29750 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:48.347436 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:48.347560 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:00:48.347646 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:00:48.348696 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:48.354509 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:48.354596 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:48.354745 29750 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:00:48.424782 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:48.426668 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:48.428355 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.430286 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.434636 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:48.437044 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:48.446091 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:48.492360 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:48.542342 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.542476 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:48.544605 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:48.551391 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.551793 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:48.556996 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:48.596437 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:00:48.633139 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:48.636091 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:48.637171 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.640648 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:48.642436 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.648454 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:48.665202 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:48.740129 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:48.742949 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:48.744158 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.751524 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:48.752346 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:48.764992 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.857049 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:48.860306 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.862547 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:48.866358 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:48.866887 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:48.896149 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:48.904708 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.908030 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:48.910804 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.917421 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:48.927925 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:48.947544 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:49.019188 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:49.021603 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.022776 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:49.022897 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:49.031845 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.032778 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:49.042894 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:49.044812 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:49.048223 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:49.093993 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:49.095973 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.104267 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:49.105556 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:49.107204 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:49.113442 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:49.114070 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:49.119647 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:49.142204 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:49.169153 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:49.203809 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.204413 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:49.204752 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.211040 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:49.212261 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:49.222640 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:49.421581 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:49.421871 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:49.428660 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.431968 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.440502 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:49.442112 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:49.457167 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:49.514632 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:49.520371 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:49.536700 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:49.540099 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:49.540531 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:49.548454 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:49.548557 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.549225 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.593866 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:49.594367 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:49.598271 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:49.602605 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:49.603745 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:49.605438 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.610847 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:49.612052 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:49.633255 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:49.667886 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:49.678925 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:49.707687 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:49.719095 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:49.722818 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.725257 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:49.733335 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:49.740665 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:49.742676 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.832080 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:49.833117 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:49.838908 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.838961 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.844110 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:49.862035 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:49.893487 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 155 similar messages]
I20240503 14:00:49.969914 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:49.969929 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:49.970265 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:49.971726 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:49.981693 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:49.982628 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:49.987982 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 14:00:49.992184 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:49.994470 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:49.996002 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.000406 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:50.012349 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:50.012885 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:50.039383 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:50.042290 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:50.053246 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:00:50.088479 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:50.111377 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:50.120930 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:50.145190 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:50.147439 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.151669 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.153944 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:50.154301 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:50.158648 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:50.166397 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:50.173799 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:50.178001 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:50.192107 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:50.195361 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.200315 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:50.208388 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:50.210992 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.225765 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:50.228664 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:50.257167 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 14:00:50.311632 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:50.337211 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:50.343695 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:50.365190 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:50.430650 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:50.501510 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:50.520207 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:50.522199 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:50.527981 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:50.532187 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.537060 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:50.538632 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.546432 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:50.573897 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.575349 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:50.576964 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:50.588351 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:50.598861 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:50.601549 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:50.604061 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:50.686812 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:50.708774 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:50.715454 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:50.717978 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.718917 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:50.719036 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.722609 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:50.728096 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:50.733990 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:50.748203 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.751394 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:50.754576 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:50.784196 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:50.787930 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.791532 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:50.868183 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:50.871403 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:50.873208 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.873592 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:50.873623 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.894708 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:50.929710 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:50.978083 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:50.980273 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:50.984441 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:50.986477 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:50.990458 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:50.995361 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:51.029518 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:00:51.031116 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:00:51.091367 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:51.091367 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.099296 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:51.111027 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:51.111667 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:51.122795 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:51.125111 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.126863 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:51.130288 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:51.143015 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:51.145236 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.148478 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:51.169497 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:51.174695 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:51.251201 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:51.251436 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:51.252586 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.256868 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.258525 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:51.277148 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:51.359326 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:51.384483 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:51.458802 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.464051 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:51.465225 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:51.466919 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:51.467856 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.472992 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:51.498977 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:51.537896 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:51.575264 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:51.584800 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:51.586231 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.594285 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:51.598608 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:51.600450 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:51.604877 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:51.609548 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:51.618161 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:51.639431 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.639695 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:51.640120 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:51.654105 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:51.666431 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.730216 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:51.733281 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:51.734685 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.744992 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:51.746076 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:51.782039 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.790817 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:51.796537 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:51.797048 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:51.799114 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.815006 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:51.821607 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:51.829859 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:51.847693 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:51.895149 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:51.899591 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:51.911918 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.913045 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:51.924405 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:51.925812 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:51.929486 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:51.940970 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:52.010936 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:52.026446 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:52.026847 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.030086 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:52.039986 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.040436 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:52.041363 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:52.119361 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:52.121062 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:52.123728 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.124025 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:52.128223 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:52.132025 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:52.132277 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:52.136750 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:52.153559 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:52.183020 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:52.203563 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:52.234844 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:52.236505 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:52.239420 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:52.240275 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:52.240499 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.241585 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.248356 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:52.335127 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.336225 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:52.337685 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:52.339433 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:52.346762 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:52.347920 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.362076 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:52.408547 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:52.448303 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:52.448494 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.451774 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.455435 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:52.456876 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:52.473131 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:52.555215 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:52.560212 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.563614 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:52.578075 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:52.578795 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:52.591372 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:52.612218 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:52.613850 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:52.624878 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:52.626255 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:52.632746 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:52.633155 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:52.633875 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:52.641963 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:52.648025 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:52.656227 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:52.659039 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:52.673297 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:52.730551 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:52.740535 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:52.743096 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:52.743990 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.752017 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:52.753825 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:52.761744 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.774019 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:52.779135 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.779850 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:52.786286 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:52.792577 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:52.795354 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:52.804697 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:52.840500 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:52.896301 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:52.911439 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:52.911468 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.911980 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:52.915407 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:52.920831 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:52.936542 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:52.944262 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:52.960304 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:53.007292 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:53.097949 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:53.102799 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:53.103747 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.106259 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:53.115010 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:53.118420 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.129575 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:53.165271 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:00:53.230662 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:53.243265 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:53.248364 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.249503 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.257618 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:53.258764 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:53.260483 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:53.270700 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:53.271658 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:53.275414 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:53.286118 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:53.295404 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:53.307394 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.316180 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:53.316336 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:53.382706 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:53.387974 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:53.419689 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.421591 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:53.421662 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.431558 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:53.441273 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:53.442181 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:53.530308 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:53.533244 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:53.536677 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:53.543395 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:53.562525 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.564224 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:53.619976 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:00:53.647918 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:53.648175 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.648537 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.650856 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:53.662631 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:53.663864 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:53.722308 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:53.724222 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:53.724231 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:53.724876 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.725703 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.737064 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:53.770700 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:53.781736 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:53.806142 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.812786 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.816022 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:53.816395 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:53.823640 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:53.844568 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:53.856835 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:53.880052 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:53.881201 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:53.881381 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:53.883208 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:53.885144 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:53.903532 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:53.986477 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:53.999042 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:54.051578 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:54.135583 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:54.137372 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:54.137854 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:54.147882 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:54.150269 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:54.168735 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:54.168871 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:54.171394 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:54.237385 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:54.237483 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:54.243664 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:54.249751 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:54.257833 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:54.263512 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:54.289240 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:54.292140 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:54.304155 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:54.313519 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:54.313616 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:54.313872 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:54.316031 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:54.317183 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:54.580850 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:54.581005 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:54.599267 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:54.613852 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:54.615653 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:54.618206 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:54.624557 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:54.634918 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:54.635679 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:54.646325 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:54.657763 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:54.695001 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:54.707775 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:54.897368 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240503 14:00:54.897917 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:54.930519 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:54.994745 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:55.008352 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:55.012706 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:55.041436 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 14:00:55.041997 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:55.084256 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:55.101519 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:00:55.153026 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:55.181139 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:55.280225 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:55.298614 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:55.309643 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:55.311882 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:55.312039 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:55.316370 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:55.316761 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:55.334792 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:55.337563 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:55.392452 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:55.422752 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:55.424870 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:55.425870 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:55.427156 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:55.447335 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:55.447465 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:55.471387 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:00:55.510358 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:55.528283 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:55.531157 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:55.534073 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:55.535806 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:55.543437 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:55.557181 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:55.565694 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:55.632375 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:55.637885 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:55.641716 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:55.641873 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:55.642037 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:55.644496 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:55.656575 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:55.691085 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:55.725059 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:55.726989 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:55.728557 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:55.729761 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:55.736508 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:55.737475 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:55.745733 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:55.765509 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:55.798478 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:00:55.930922 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:55.959182 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:55.960270 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:55.962174 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:55.967543 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:55.974897 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:55.991845 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:56.055034 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:56.064047 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.064352 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:56.067595 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:56.076747 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:56.076948 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:56.080412 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:56.111752 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:56.133520 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:56.133920 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:56.149986 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.151082 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.162312 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:56.163362 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:56.228540 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:56.228993 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.231756 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:56.233098 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:56.240439 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.245926 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:56.277832 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:56.298810 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:56.299813 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:56.300315 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.303978 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:56.306633 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:00:56.313586 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:56.326622 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:56.380115 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:56.413575 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:56.414530 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:56.416836 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.418213 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:56.419203 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:56.421737 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.480904 29295 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:00:56.481030 29295 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:00:56.481441 29295 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:00:56.483196 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:56.483640 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:56.483723 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:56.483906 29295 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:00:56.538317 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:56.539248 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:56.546452 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:56.547314 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:56.550832 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:56.553644 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.561544 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.577333 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:56.617890 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.617869 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:56.618412 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:56.621308 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:56.622658 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:56.633019 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:56.633860 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:56.634724 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:56.713254 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:56.717945 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:56.718518 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:56.719197 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:56.719422 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.720912 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:56.721778 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.723402 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:56.783092 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:56.873454 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:56.876518 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:56.877367 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:56.878518 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.884603 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:56.885951 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.974531 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:56.976128 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:56.978658 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:56.984608 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:56.986219 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:56.987061 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:57.043493 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:57.060639 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.064872 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:57.066191 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:57.080686 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:57.080976 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.096246 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:57.104630 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:57.117627 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:57.126814 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:57.130363 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:57.131009 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:57.131678 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.131803 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:57.134771 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:57.135255 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:57.137058 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:57.223012 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:57.223443 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:57.236081 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:57.239259 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:57.248862 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:57.254985 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:57.255157 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:57.257025 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.260592 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:57.262825 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:57.274564 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.284489 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:57.290659 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:57.295729 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.298233 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:57.301903 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:57.313608 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:57.389335 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:57.389899 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:57.393900 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:57.396507 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.410861 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:57.414449 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:57.481803 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:57.507818 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:57.543480 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:57.557511 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:57.561591 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.563572 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:57.563925 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:57.564208 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:57.564951 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:57.567018 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:57.672046 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.673506 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:57.677484 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:57.679499 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:57.685127 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:57.687678 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:57.695751 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.739797 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:00:57.742930 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:57.744704 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:57.746368 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:57.747020 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.747678 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.748320 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:57.779695 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:57.817435 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:57.844900 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:57.847517 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.855144 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.855697 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:57.856144 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:57.857877 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:57.878772 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:57.890072 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:57.890718 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:57.950157 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:57.951010 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:57.952777 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:57.955121 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:57.956826 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:57.984987 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.025879 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.026512 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:58.029781 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:58.031318 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:58.034518 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:58.040944 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:58.055881 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.121543 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:58.122038 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.123088 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:58.131439 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:58.131958 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.132280 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:58.139043 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:58.157315 978 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:00:58.157414 978 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:00:58.157780 978 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:00:58.158563 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:00:58.158674 12036 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:00:58.158730 12036 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:00:58.159565 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:58.159720 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:58.159772 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:00:58.159998 978 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:00:58.199108 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:58.202983 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:58.206544 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:58.215266 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.217482 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:58.222544 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:58.240705 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:58.257120 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:58.269639 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:58.270540 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:58.274749 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.277871 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.280936 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:58.283591 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:58.316181 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:58.367483 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:00:58.400995 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:58.403093 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:00:58.529455 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:58.531174 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:58.531181 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.534775 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:58.547652 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:58.560437 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.612820 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:58.613770 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.616794 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:58.625825 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:58.639711 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.644785 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:58.693681 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:58.694728 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:58.697142 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:58.699383 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:58.705353 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.713917 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:58.721909 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:58.741787 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:58.771939 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:58.775542 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:58.777593 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:00:58.782866 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:58.782924 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.786566 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:58.791940 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.884687 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:58.884691 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:58.889487 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:58.889905 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:58.890630 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:58.891404 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:58.907420 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:00:58.919623 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:58.932044 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:58.959339 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:58.997861 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:58.998461 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:59.004135 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.010083 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:59.011816 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:59.015664 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.058507 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:59.063192 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.064534 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:59.069854 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:59.073562 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.083400 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:59.147770 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:59.152567 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:59.158973 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.161381 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:59.169584 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.171707 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:59.197057 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:59.204917 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:59.208535 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:59.210983 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.213970 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.232635 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:59.241930 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:00:59.265165 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:00:59.311174 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:59.332650 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:59.446791 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:59.498813 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:00:59.529680 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:59.532179 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:59.535878 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.541895 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.542098 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:59.544158 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:59.563771 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:59.564064 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.571888 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.591251 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:59.593777 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:59.610016 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:59.622733 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:59.626900 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:00:59.671680 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:00:59.681777 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:59.682251 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:59.685075 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.685706 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:00:59.693766 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:59.715854 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:59.719060 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:59.725986 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:59.728852 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:59.732504 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:59.738162 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.746187 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:00:59.756428 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:59.760908 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:59.829800 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:00:59.838531 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:00:59.840391 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:00:59.840682 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:59.841307 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.844676 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:00:59.847406 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.853258 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:59.897714 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:00:59.943578 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.944516 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:00:59.948442 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:00:59.948714 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:00:59.950471 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:00:59.954123 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 162 similar messages]
I20240503 14:00:59.972486 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:00.007553 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:01:00.046190 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:01:00.048916 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:00.054030 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:00.054410 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.059484 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:00.061074 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:00.063728 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:00.088089 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:01:00.103384 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 14:01:00.103906 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:00.104140 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:00.105862 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:00.106351 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.108815 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:00.112591 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.137903 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:00.152114 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 14:01:00.206173 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:00.207273 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:00.208299 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.213277 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:00.220932 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.228904 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:00.262215 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:00.278447 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:01:00.342800 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:00.366063 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:01:00.393632 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:01:00.426890 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:00.428246 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:00.435360 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:00.438182 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.442214 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:00.449690 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:00.454905 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:01:00.512029 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:00.516314 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:01:00.531651 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:00.532903 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:00.534497 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:00.538601 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:00.557422 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 14:01:00.564369 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:00.569954 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:01:00.581617 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.586473 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:00.590870 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:00.603641 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.612159 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:00.622793 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:00.634991 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.718310 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:00.719385 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.723963 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:00.732683 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:00.734854 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:00.742077 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.765403 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:00.792572 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:00.793678 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:00.796723 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.799284 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.801707 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:00.817687 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:00.824563 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:00.857450 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:01:00.885711 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:01:00.911995 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:00.912923 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.915516 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:00.917412 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:00.935706 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:00.944003 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:00.958132 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:01.016535 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:01:01.016638 29295 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:01:01.016742 29295 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:01:01.017169 29295 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:01:01.018906 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:01.019486 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:01.019557 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:01:01.019696 29295 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:01:01.037890 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:01.039096 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:01.049307 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.050626 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:01.056819 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:01.059384 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.121857 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:01.122547 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:01.127684 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.128489 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.129266 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:01.136797 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:01.148986 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:01.239984 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:01.241844 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:01.243021 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.243659 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.248136 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:01.267429 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:01.357699 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.363970 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:01.365053 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:01.373781 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:01.375357 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:01.379112 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:01.390645 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:01:01.391726 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:01.405718 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:01.454602 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:01.483268 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:01.489126 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:01.489544 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.490792 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:01.513089 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:01.515621 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:01.553236 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:01:01.572928 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:01.597028 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:01.603094 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.608846 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.617487 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:01.617575 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:01.625176 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:01.653038 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:01.654873 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:01.665448 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.665889 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.673990 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:01.675098 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:01.681829 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:01.728149 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:01.785912 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.786442 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:01.799206 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:01.800810 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.801789 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:01.829856 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:01.837960 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:01.849409 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:01.854748 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.856029 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:01.856312 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:01.863708 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:01.865800 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:01.886505 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:01.960453 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:01.963783 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:01.965224 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:01.966003 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:01.969585 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:01.972009 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:01.975739 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:02.000490 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:02.093935 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:02.100492 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:02.127969 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:02.133690 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.136611 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:02.137565 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.149426 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:02.150453 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:02.193186 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:02.275461 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:02.279167 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:02.286676 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:02.304591 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.306919 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:02.317203 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:02.341581 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.346297 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.360177 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:02.362464 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:02.367292 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:02.380705 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:02.393071 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:02.393993 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:02.427742 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:02.431610 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:01:02.499601 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:02.522454 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:02.525611 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:02.528620 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:02.529332 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:02.529417 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.534235 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.550355 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:02.570127 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:01:02.575181 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:01:02.584095 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:01:02.631880 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:01:02.641412 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:02.647979 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:02.650270 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.651258 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:02.652316 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:02.652539 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.668432 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:02.694291 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:02.750424 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.750883 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.756088 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:02.759418 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:02.763970 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:02.776259 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:02.827658 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.829303 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:02.836630 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:02.837013 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:02.838301 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.839519 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:02.865551 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:02.916716 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:02.917114 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.917423 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:02.921079 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:02.921311 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:02.941490 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:02.959851 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:01:02.971491 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:03.006596 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:03.009743 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.009757 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:03.011646 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:03.024133 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:03.028322 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:03.028740 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:03.048125 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:01:03.175454 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:01:03.235576 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:03.280540 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:03.281630 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:03.286056 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:03.286957 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:03.289834 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:03.303400 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:03.309420 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.324752 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:03.325109 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:03.398237 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.402698 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:03.403107 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:03.411810 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:03.419726 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.420281 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:03.470094 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.471295 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:03.479878 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.481977 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:03.483892 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:03.498296 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:03.505674 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:03.566308 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:03.566594 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:03.569900 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.571941 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:03.579066 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:03.598481 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:03.621961 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:03.681958 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:03.685312 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:03.685832 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:03.693046 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.699002 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:03.704185 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:03.779299 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:03.814976 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:01:03.817006 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.817039 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:03.817313 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.829941 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:03.836774 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:03.841953 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:03.866691 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:03.867403 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:03.867684 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.872380 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.877110 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:03.890285 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:03.902917 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:03.990269 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:03.997804 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:03.998356 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:04.002810 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.013787 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:04.020742 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:04.022014 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:04.023993 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:04.030777 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:04.031606 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:04.042131 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:04.051980 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.054491 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:04.055546 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:01:04.084173 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:04.085614 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:01:04.259263 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:01:04.268863 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:04.364298 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:04.381944 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:04.383462 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:04.390519 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.393893 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:04.396073 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.399374 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:04.404850 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:04.408854 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.409955 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:04.411074 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:04.413030 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.415377 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:04.439726 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:04.456697 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:04.527952 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:04.528412 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:04.568498 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:04.571336 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:04.571741 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:04.576928 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:04.579105 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:04.579634 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.580341 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:04.583918 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:04.600471 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:04.614114 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.623200 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:04.626749 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:04.634796 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:04.640981 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.732972 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:04.736574 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.738415 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:04.739255 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:04.739893 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:04.751554 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.752699 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:04.771051 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:04.824647 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:04.848589 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:04.857744 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:04.863946 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:04.864385 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:04.874549 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:04.887804 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:04.946832 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:01:04.961153 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:04.990610 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 158 similar messages]
I20240503 14:01:04.992154 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:04.995342 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:05.000550 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.008320 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:05.009197 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.014557 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.024459 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:05.031105 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.031132 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:05.035132 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:05.037565 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:05.040515 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:01:05.047540 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:05.049336 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:05.070674 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:01:05.072325 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:01:05.084834 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:01:05.143983 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:05.145247 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:05.150027 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:05.151322 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:05.153689 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.159603 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.185190 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:05.329847 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:05.355141 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:05.360870 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 14:01:05.361900 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:05.363296 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:05.365226 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:05.372299 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 14:01:05.373193 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:05.380223 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:05.399750 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:05.401957 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:05.424739 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:05.456511 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:05.503077 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:05.507017 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.510185 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.532644 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:05.534121 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:05.534315 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:05.547839 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:05.585961 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:05.604084 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:05.606549 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:05.608318 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.616268 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.639577 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:05.645222 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:01:05.649502 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:05.677465 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 14:01:05.726285 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:05.731740 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:05.733530 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.745015 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:05.746459 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.747224 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:05.806548 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:05.808028 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.808372 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:05.809084 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:05.816735 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:05.818310 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:05.821401 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.914626 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:05.916505 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:05.918807 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.919163 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:05.944937 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:05.949160 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:06.016384 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:06.020469 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:06.042824 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:06.044497 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:06.047600 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:06.048453 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.063582 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.068344 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:06.116199 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:06.117362 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:06.118592 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:06.124754 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:06.124764 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:06.125005 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.134335 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:06.210354 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:01:06.231892 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:06.233603 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.233985 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.235440 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:06.240516 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:06.250231 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:06.260596 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:06.373203 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.377528 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:06.379434 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.384824 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:06.390609 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:06.392285 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:06.404812 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:06.486065 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:06.487877 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.494032 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:06.496148 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.504011 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:06.507570 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:06.529222 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:01:06.581823 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:06.593245 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:06.596029 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.603200 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:06.604076 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:06.607579 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.609413 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:06.617172 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:06.631769 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:01:06.673678 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:06.682832 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:06.684612 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.687225 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:06.687330 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.692034 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:06.701205 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:06.713675 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:06.755290 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:06.772989 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:06.781332 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.781654 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:06.782146 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:06.795008 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:06.804986 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:06.807150 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:06.859010 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:06.860256 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:06.863569 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.864410 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:06.869738 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:06.870942 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:06.938624 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:06.975909 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:06.976456 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:06.976941 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:06.981580 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:06.987370 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:07.009214 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:07.028818 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:01:07.056444 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:07.092326 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:07.093042 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:07.093581 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.097414 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:07.112970 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:07.116982 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:07.159484 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:07.204510 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.206647 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:07.209174 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.213424 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:07.219883 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:07.220664 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:07.236274 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:07.263989 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:01:07.294548 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:07.309458 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:07.309618 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:07.316772 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:07.324064 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.325505 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.331404 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:07.335683 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:01:07.372948 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:07.426074 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:07.431353 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.431974 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:07.434687 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:07.437870 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:07.438580 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:07.452044 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:07.454667 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:07.552937 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:07.553243 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:07.560360 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.561156 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:07.562716 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.576367 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:07.577342 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:07.592128 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:07.593914 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:07.598604 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:07.607100 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:07.607146 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.624514 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:07.634586 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:01:07.722616 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:07.739349 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:07.741758 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.749085 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:07.749326 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:07.750721 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.758466 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:07.761279 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:07.765566 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:07.767899 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:07.773627 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:07.782153 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:07.796118 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.796996 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:07.797233 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:01:07.826354 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:07.894366 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:01:07.896704 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:07.903985 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.904673 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:07.904677 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:07.909945 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:07.911399 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:07.913870 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:07.924176 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:01:07.978837 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:01:08.095105 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:08.099720 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:08.103516 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.103875 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:08.106070 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:08.108615 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.109899 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:08.177850 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:08.238291 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:08.238729 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:08.240818 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.242130 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:08.243919 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:08.246016 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.248844 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:08.252499 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:08.271411 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:08.285820 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:08.286804 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.291712 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.291981 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:08.303457 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:08.390323 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:08.402343 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:08.404219 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:08.404749 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:08.405373 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:08.405761 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:08.416623 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.418836 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.511889 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:08.516175 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.516666 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.517423 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:08.523200 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:08.529831 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:08.534109 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:08.569936 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:08.640300 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:08.641855 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:08.644873 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:08.646526 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.662961 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:08.672575 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.680150 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:08.681568 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.684098 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.685312 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:08.694991 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:08.697715 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:08.719154 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:08.815083 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:01:08.828550 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:08.835606 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:08.842787 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:08.843072 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.844904 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.845634 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:08.854293 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:08.862574 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:08.865100 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:08.874699 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:08.885789 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:08.898315 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:08.932581 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:09.020053 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:09.082262 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:01:09.163545 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:01:09.197194 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:09.203801 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:09.210551 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:09.211550 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:09.212070 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:09.215011 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:09.225095 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:09.227200 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:09.228394 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:09.246495 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:09.248658 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:09.264544 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:09.264686 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:09.267328 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:09.277598 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:09.301384 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:09.329514 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:09.359591 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:09.385797 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:09.392169 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:09.394830 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:09.396553 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:09.402807 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:09.404441 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:09.411585 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:09.454887 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:09.457522 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:09.460424 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:09.464107 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:09.466248 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:09.470072 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:09.519332 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:09.570714 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:09.574616 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:09.574863 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:09.575102 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:09.575345 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:09.592639 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:09.624908 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:01:09.634534 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:01:09.676069 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:09.693056 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:09.693270 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:09.695678 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:09.700889 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:09.706897 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:09.714133 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:09.774914 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:09.777667 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:09.791059 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:09.793788 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:09.793885 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:09.807412 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:09.809230 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:09.863781 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:09.869295 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:09.870023 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:09.873576 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:09.874245 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:09.899109 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:09.930084 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:09.945180 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:09.946393 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:09.951453 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:09.953915 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:09.954612 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:09.955018 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:09.959403 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:01:09.971298 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:09.981969 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:10.005406 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240503 14:01:10.005968 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:01:10.140648 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:01:10.182768 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:01:10.192560 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:10.194834 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.195869 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:10.197826 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:10.198082 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:10.207917 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.216205 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:10.280086 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:10.301760 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.309334 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:10.309513 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:10.310402 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.311091 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:10.322396 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:10.360204 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:10.388046 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.389271 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:10.390151 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:10.400204 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:10.401700 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:10.402946 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:10.442592 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:01:10.452364 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:01:10.453070 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:10.455817 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:01:10.481726 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:10.499464 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:10.500011 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:10.502795 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:10.507364 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:10.508289 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:10.515179 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:10.539888 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.564422 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:10.566558 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:10.570641 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.574623 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.581933 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:10.594369 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:10.609288 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:01:10.634783 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:10.685482 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.690407 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:10.691412 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:10.694689 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:10.696249 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:10.699697 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:10.753484 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:10.757701 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:01:10.764988 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:10.808024 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:10.809492 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:10.812420 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.818866 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.819836 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:10.848889 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:10.893199 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:10.894088 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:10.897570 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.900817 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:10.914634 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:10.917452 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:10.919040 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:10.978133 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:11.004499 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:11.009801 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.011174 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:11.014065 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.024394 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:11.030529 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:11.044771 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:11.135959 5971 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:01:11.136080 5971 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:01:11.136487 5971 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:01:11.136911 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:11.137398 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:11.137632 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:11.137864 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:11.137935 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:01:11.138803 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.139178 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.139259 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:11.140838 5971 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:01:11.150763 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:11.162256 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:11.170044 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:11.248301 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:11.250712 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:11.250979 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.252723 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:11.255867 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.275887 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:11.334079 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:01:11.343207 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:11.347028 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.349520 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:11.354280 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.357344 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:11.357836 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:11.432718 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.435595 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:11.437510 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:11.438117 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:11.444301 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.467134 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:11.503922 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:11.510854 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:01:11.526093 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:11.529345 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.531728 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:11.539942 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:11.551802 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.563539 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:11.582867 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:11.586745 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:11.606022 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:11.610947 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.611397 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:11.615406 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:11.619939 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:11.631415 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:11.642994 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:01:11.718642 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:11.721520 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.721808 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:11.727280 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.728796 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:11.735801 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:11.751039 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:11.755615 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:11.755887 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:11.758076 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:11.786217 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:11.863524 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:11.863544 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.864636 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:11.866206 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:11.866317 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:11.890552 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.985105 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:11.986276 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.987939 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:11.988593 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:11.990446 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:12.000478 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:12.008236 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:12.067694 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.071050 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:12.071470 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:12.072368 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.074564 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:12.079391 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:12.092742 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:12.125595 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:12.176792 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:12.182169 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:12.183543 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:12.185123 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.185869 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.186234 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:12.207279 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:12.298156 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:12.299809 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:12.301599 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.304709 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:12.308866 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:12.309437 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:12.362444 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:12.364265 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:12.368831 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:01:12.369520 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:12.370796 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:12.373921 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:12.378376 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.378389 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:12.384009 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:12.385546 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:12.441287 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:12.479192 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:01:12.480481 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.484061 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:12.490725 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:12.499470 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:12.501004 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.501367 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:12.532631 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:12.542038 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:12.542371 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:12.546352 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:12.546824 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:12.549140 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.550196 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:12.572921 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:01:12.643585 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:12.646227 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.646977 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:12.648398 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.655654 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:12.656877 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:12.674149 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:12.713333 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:12.741179 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:12.766862 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:12.810135 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:12.813165 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:12.880766 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:12.882418 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:12.888475 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:12.889549 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:12.892128 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:12.911931 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:12.923889 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:12.992054 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:12.994969 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:12.997248 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.000450 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:13.004864 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.018760 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:13.050141 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:13.052845 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.056105 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:13.059502 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:13.074788 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:01:13.077039 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.078634 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:13.157807 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:13.158787 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:13.162418 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:13.164726 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:13.166307 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.180413 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:13.191627 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:13.282688 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:13.283555 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:13.284374 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.287348 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:13.290189 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:13.292987 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:01:13.293931 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:13.300200 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:13.329162 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:13.353921 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:13.415555 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:13.421669 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:13.423621 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.433936 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.434505 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:13.436448 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:13.461012 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:13.465809 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:13.467381 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:13.468586 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.470278 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:13.482633 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:01:13.495961 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:13.576529 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:01:13.588624 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.589071 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:13.589617 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:13.592741 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:13.593657 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:13.595871 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:13.631772 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:13.643453 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.644068 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:13.645004 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:13.653522 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:13.655045 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:13.675413 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:13.815393 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:13.908838 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:13.989409 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:13.990995 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:13.994658 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:13.995165 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.004146 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:14.004582 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.007093 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:14.007226 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:14.022931 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:14.031131 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:14.031878 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:14.055423 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.056846 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:14.061319 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:01:14.084563 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:01:14.099282 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:01:14.106338 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:01:14.127472 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:01:14.133469 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:14.151677 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:14.155092 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:14.158974 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:14.159788 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.169400 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:14.169556 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:14.187014 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:14.200245 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:14.242648 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:14.244262 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:14.244959 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.245978 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.248513 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:14.250010 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:14.313377 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:14.353997 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:14.355669 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.357116 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:14.358793 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:14.359925 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:14.363480 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:14.406080 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:14.438699 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:14.452380 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:14.480163 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:14.485623 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:14.488516 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.490938 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.499061 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:14.502576 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:14.536674 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:14.554148 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:14.555786 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.558367 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.563395 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:14.566071 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:14.601763 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:14.650463 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:14.652823 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.655130 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:14.660060 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.662789 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:14.664640 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:14.679352 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:14.733246 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.733773 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:14.734407 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:14.739943 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.749660 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:14.753543 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:14.800436 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:01:14.821457 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:14.836879 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:01:14.884819 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:14.972240 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:14.974640 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:14.984344 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:14.988965 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:14.994380 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:15.015022 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:15.035615 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240503 14:01:15.037237 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:01:15.061486 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:15.082688 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:15.086462 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.095405 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.097386 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:15.107493 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:01:15.113132 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:15.113245 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:15.118638 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:15.133179 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:15.150442 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.151432 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:15.158300 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:15.163543 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.169420 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:15.182073 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:15.267299 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.271216 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.271868 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:15.279011 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:15.286211 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:15.295903 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:15.314771 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:01:15.320848 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:15.327045 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:15.330749 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.333691 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:15.333992 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.336529 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:15.342252 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:15.360672 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:15.383896 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:01:15.430559 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:15.451409 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:15.453617 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:15.454782 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:15.456326 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.458986 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:15.459837 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:15.486860 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:15.497786 7787 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:01:15.497910 7787 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:01:15.498509 7787 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
W20240503 14:01:15.498987 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:01:15.499205 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:01:15.499348 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:01:15.499420 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:01:15.499677 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:15.500077 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:15.500151 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:01:15.500290 7787 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:01:15.549881 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:01:15.552703 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:15.561718 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:15.561939 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.566412 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:15.568516 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:15.572824 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:15.593008 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:15.596361 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:15.654444 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:15.660466 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:15.663388 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.665081 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.679783 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:15.685931 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:15.778309 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:15.778589 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.783406 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:15.785161 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:15.787837 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.805536 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:15.822827 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:01:15.843901 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:15.869297 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:15.884835 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:15.885268 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:15.886760 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:15.886849 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:15.891556 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:15.904191 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:15.915403 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:15.920395 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:01:15.948038 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:16.004993 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:16.008162 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.008855 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:16.013561 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:16.025249 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:16.025966 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:16.027498 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.106351 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:16.110668 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.111182 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:16.113723 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.137643 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:16.138942 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:16.183122 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:16.183718 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:16.187444 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:16.189191 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.196916 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:16.197405 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:16.229578 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:16.287392 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:16.289901 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:16.293694 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.302948 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.307693 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:16.310330 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:16.348274 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:16.354478 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.355638 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:16.359628 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:16.372025 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.375002 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:16.381173 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:16.399502 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:16.402563 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:16.437032 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:16.438323 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:01:16.481294 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:16.481933 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.483726 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.484150 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:16.491900 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:16.506208 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:16.516465 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:16.590665 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:16.629616 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:16.630097 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.632509 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.639833 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:16.647408 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:16.655387 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:16.709611 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:16.734829 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:16.756527 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:16.757499 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:16.759577 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:16.760021 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:16.760257 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.778965 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:16.804669 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:16.830525 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:16.835024 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:16.836645 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.837174 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:16.841138 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.854727 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:16.857743 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:16.913590 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:16.949169 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:16.950028 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.952260 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:16.952411 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:16.956350 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:16.965432 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:16.984334 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:17.032450 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:17.074079 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.077426 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.082432 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:17.085198 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:17.086475 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:17.088840 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:17.099349 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:17.118836 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:17.120860 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:17.123544 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:17.131556 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.137389 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:17.140246 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.171511 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:17.257200 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:17.259881 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.263404 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:17.267935 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:17.269412 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:17.270418 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:17.292101 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:17.312356 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:17.327901 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.328648 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:17.329993 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:17.335429 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:17.339630 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:17.362529 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:17.388610 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:17.428604 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:17.428604 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:17.430967 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:17.434754 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:17.441617 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:17.444849 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:17.449676 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:17.449905 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:17.480854 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:17.595979 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:17.642349 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:17.648425 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:17.649413 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:17.650728 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.655651 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:17.661751 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:17.663420 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:17.708470 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:17.754921 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:17.760201 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:17.761763 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.770823 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:17.780004 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.807490 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:17.813761 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:17.818769 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.821425 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:17.836449 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:17.843397 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.843740 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:17.863276 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:17.886226 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:17.926072 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:17.928805 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:17.930456 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:17.933367 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:17.935179 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:17.942087 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:17.943033 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:18.025627 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:18.044991 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:18.045028 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.050396 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:18.050496 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:18.053299 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:18.067565 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:18.090312 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:18.096937 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:18.106786 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:18.118867 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:18.168706 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.173754 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:18.174137 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.175683 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:18.180039 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:18.189121 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:18.208906 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:18.214241 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.221398 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.222489 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:18.232323 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:18.249830 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:18.313306 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:18.324508 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:18.350401 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:18.352974 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:18.362056 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:18.366295 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:18.369100 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:18.369927 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.378275 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:18.380640 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.392714 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.392959 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:18.395776 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:01:18.410547 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:18.432384 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:18.445647 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:18.506417 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:18.648923 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:01:18.694136 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:18.719756 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:18.720328 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:18.735322 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.736007 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:18.756192 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.763989 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:18.782953 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:18.783703 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.784953 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:18.785122 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.790484 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:18.814276 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:18.871793 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:18.901289 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:18.917048 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:18.919092 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:18.922946 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.927731 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:18.937016 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.941026 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:18.956220 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:18.966773 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:18.969851 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:18.972558 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:18.974844 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:18.979697 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:18.995721 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:19.025645 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:19.087472 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:19.088768 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:19.094341 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.095590 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:19.099417 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.119801 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:19.130949 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:01:19.134908 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:01:19.158071 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:19.192008 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:19.204970 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:19.209610 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.212388 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.214035 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:19.216512 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:19.220167 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:19.289016 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:01:19.303462 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:19.307536 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:19.310601 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.313014 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:19.326860 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:19.334829 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:19.336771 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:19.338333 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:19.389482 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:19.396692 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.400578 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:19.402101 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:19.423485 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.427515 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:19.450852 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:19.480724 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:19.482084 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:19.484189 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:19.486418 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:19.496244 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.497129 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.497682 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:19.545075 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:01:19.559734 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:01:19.687847 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:19.727752 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:19.727744 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.729650 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:19.730628 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:19.734042 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:19.747831 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:19.759656 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:19.803594 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:19.833125 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:19.842229 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.847412 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:19.849329 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.876364 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:19.885353 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:19.895829 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:19.899006 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:19.910799 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.912837 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:19.917625 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:19.928642 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:19.939945 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:19.941843 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:20.025257 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:20.025660 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:20.027668 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:20.028453 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:20.036480 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:20.044178 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240503 14:01:20.044857 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:20.044950 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:20.045418 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:20.053617 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:20.079531 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:20.087340 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:20.092389 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:20.093024 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.093655 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:20.109957 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:20.110246 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:20.173056 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:20.200996 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:20.204917 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:20.208671 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.211786 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:20.212342 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:20.212780 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:20.229226 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.323438 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:20.324527 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.326893 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:20.328346 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:20.335363 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:20.335400 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.337988 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:20.425645 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.426678 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:20.427106 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:20.427675 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:20.434237 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:20.441612 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.536319 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:20.538739 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.542248 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.543466 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:20.545593 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:20.552035 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 14:01:20.552717 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:20.554776 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:01:20.562032 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:01:20.565048 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:20.579058 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:20.612429 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:01:20.622707 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:20.667822 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:20.668259 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.670101 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:20.672708 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.673420 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:20.683485 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:20.692150 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:20.729498 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:20.788520 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:20.792891 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:20.793226 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.801273 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.814051 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:20.815738 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:20.859179 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:01:20.874508 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:20.925038 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:20.926076 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.927318 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:20.929432 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:20.933696 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:20.947530 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:20.973273 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.974232 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.974711 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:20.974920 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:20.978209 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:20.992600 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:21.020886 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:21.121291 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:21.126171 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:21.130056 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.141129 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:21.141718 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:21.141744 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:21.143368 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:21.144666 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:21.148130 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.151417 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:21.157219 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.162724 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:21.184837 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:21.237422 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:01:21.271984 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:21.289746 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:21.290081 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.290428 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:21.297011 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.298588 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:21.300042 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:01:21.304287 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:21.367797 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:21.372838 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:01:21.404574 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:21.484194 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:21.484586 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:21.489908 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.489989 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:21.512913 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:21.513051 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:21.514457 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:21.572052 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:21.572054 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:01:21.634163 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:21.635933 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.636009 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:21.641148 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:21.642921 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:21.643414 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:21.648574 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:21.652740 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.660933 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.662617 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:21.672657 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:21.679662 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:21.680981 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:21.682215 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:21.784436 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:21.791417 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:21.792968 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:21.802263 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.803906 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:21.804340 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.823113 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:21.836369 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:21.846465 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:21.864317 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:21.887746 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:21.909248 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:21.912886 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:21.915210 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.916779 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:21.925431 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:21.925762 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:21.984309 10204 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:01:21.984447 10204 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:01:21.984915 10204 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:01:21.986434 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:21.986518 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:21.986595 12103 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:01:21.986733 10204 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:01:22.043066 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:22.048072 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:22.056618 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:22.070487 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:22.074227 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.074776 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.079977 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:22.085322 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:22.088966 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.091465 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:22.096197 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:22.098377 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.129904 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:22.134927 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:22.239802 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:22.246491 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.249486 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:22.251654 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.257412 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:22.262842 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:22.263923 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:22.264075 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:22.264302 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:22.264622 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:22.266099 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:22.295614 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:22.304349 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:22.323477 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:22.342942 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:22.367496 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:22.409315 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:22.427800 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:22.462364 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:22.479833 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:22.613867 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:22.630002 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.634073 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:22.648101 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:22.649971 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:22.652005 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:22.654995 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.658540 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:22.659442 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.662981 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:22.685853 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:22.694245 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:22.702921 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:22.788108 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:22.802970 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:22.804862 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:22.805608 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.810703 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:22.831508 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:22.857439 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:22.861781 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:22.867065 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.869933 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.870285 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:22.870491 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:22.876860 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:22.904027 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:22.918833 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:22.936965 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:22.942365 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:22.942819 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:22.985962 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:22.989333 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:22.989349 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:22.990481 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:22.999333 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:22.999747 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.100967 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:23.102747 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:23.105434 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.106410 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:23.106768 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:23.107448 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.108922 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:23.115942 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:23.211627 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:23.215334 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.215641 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.217339 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:23.232117 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:23.238991 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:23.289158 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.291162 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:23.294204 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:23.295053 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.295540 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:23.310621 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:23.390583 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:23.391408 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.394356 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.400413 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:23.404327 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:23.406966 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:23.410290 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:23.425626 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:23.430034 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:01:23.548993 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:23.609251 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:23.609997 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:23.610266 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.610561 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:23.632305 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.634258 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:23.640431 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:23.641808 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:23.728868 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:23.730808 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:23.737718 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.739564 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:23.745064 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:23.745894 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:23.783246 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:23.785843 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:23.788090 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:23.792356 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:23.792968 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.816257 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.834292 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:23.884553 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:01:23.889787 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:23.912678 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:23.915917 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:23.917490 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.925352 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:23.930891 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:23.951956 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:23.980252 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:01:23.981021 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:23.987341 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.989406 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:23.989816 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:23.997802 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:24.001022 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:24.002599 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:24.013736 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:01:24.014057 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:24.062747 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:24.097041 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:24.099879 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.107451 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:24.115447 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:24.116770 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:24.121266 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:24.128151 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.212026 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:24.212643 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:24.221762 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.222849 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:24.237069 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:24.243537 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.335008 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:24.335068 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:24.339466 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:24.343654 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.345085 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:24.346823 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:01:24.347318 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:01:24.361639 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:24.371850 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:24.402283 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:24.413414 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:24.462442 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.465319 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.465912 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:24.468042 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:24.468811 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:24.470496 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:24.535343 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:24.541756 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:24.565191 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:24.569381 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:24.570914 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:24.572135 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:24.576277 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.576519 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.684433 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:24.686583 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:24.689842 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:24.691373 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:24.697739 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.699224 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:24.734337 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:24.805855 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.811477 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:24.818292 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.823421 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:24.833614 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:24.836357 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:24.853122 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:24.862458 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:24.863777 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.873901 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:24.880981 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:24.890663 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:24.892303 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:24.923928 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:01:24.927934 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:24.989663 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:24.994268 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:24.994697 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:24.996074 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:24.996536 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:24.997686 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.004855 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:25.007274 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:25.012763 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:25.022071 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:25.028070 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:01:25.036082 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:25.056869 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:25.058916 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:25.062194 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 160 similar messages]
I20240503 14:01:25.064075 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.068287 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:25.072232 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:25.080178 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:25.174522 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:25.175333 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:25.177240 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.185979 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:25.187181 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.191001 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:25.260075 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:25.316599 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:25.317026 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:25.321942 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:25.322760 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.327401 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:25.327790 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.331384 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:25.390970 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:25.441474 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:25.444360 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:25.446082 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:25.450603 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:25.457520 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:25.459766 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:25.471284 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:25.477964 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:25.505928 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:25.527951 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:25.529572 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:25.534351 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.537554 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.541136 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:25.553315 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:25.597288 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:01:25.597994 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:25.642997 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:25.645399 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:25.648346 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.648723 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:25.651436 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:25.654096 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:25.669035 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:25.769721 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:25.783121 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.787986 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:25.791466 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:25.793342 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:25.796895 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:25.804359 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.809809 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:25.833737 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:25.836337 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:25.844046 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.847579 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:25.861467 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:25.884181 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:25.915462 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 14:01:25.953465 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:25.960512 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.968374 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:25.969455 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:25.969520 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:25.978608 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:26.006624 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:26.007861 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:01:26.024324 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:26.029590 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.030858 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:26.032021 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.034114 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:26.047605 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:26.070055 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:01:26.136124 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:01:26.139380 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:26.142520 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:26.147754 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.147773 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.149845 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:26.158202 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:26.164767 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:26.181043 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:01:26.270069 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:26.300956 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:26.311405 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:26.316396 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:26.317062 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.317270 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:26.319784 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.323105 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:26.434239 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:26.435909 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:26.437301 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:26.443387 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.443595 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.459777 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:26.478722 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:01:26.482810 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:26.513998 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:26.514719 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:26.518378 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.520359 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:26.520738 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:26.522123 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:26.544384 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:01:26.580951 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:01:26.612169 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:26.628206 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.629820 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:26.629927 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.630612 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:26.643765 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:26.650616 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:26.687667 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:26.748696 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:26.753225 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:26.756220 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.764019 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.767395 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:26.770328 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:26.779964 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:26.785821 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:26.787490 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:26.808679 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:26.813777 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.825157 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:26.830446 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:26.832504 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:26.835752 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:26.917002 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:26.926780 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.930909 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:26.932596 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:26.932713 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:26.939499 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:26.995966 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:27.001816 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:27.004895 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:27.004945 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.010505 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.025015 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:27.046139 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:27.084298 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:27.085660 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:27.086359 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:27.087018 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.088318 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:27.100060 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:27.132377 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:01:27.205008 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:27.281337 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:01:27.301579 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:27.314514 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:27.314594 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:27.316962 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.317425 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:27.323699 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:27.328498 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:27.343684 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:27.355448 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:27.402541 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:27.419629 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:27.420102 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:27.423710 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.424758 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:27.429530 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:27.435698 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:27.436215 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:27.462090 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:01:27.468127 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:27.489101 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:27.494310 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:27.496280 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.503376 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:27.505427 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:27.508383 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:27.532806 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.567386 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:27.571908 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:27.573542 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.575883 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:27.593593 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.594434 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:27.639878 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:27.643759 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:01:27.660707 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:27.681800 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.684796 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:27.684990 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:27.688400 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:27.693282 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:27.693567 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:27.716555 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:27.793399 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.797601 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:27.799765 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:27.800834 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:27.803161 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:27.805604 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:27.825448 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:27.867885 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:27.868124 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:27.868952 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.872602 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:27.891117 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:27.894343 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:27.917124 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:27.965332 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:27.969882 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:27.970490 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:27.987701 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:27.995863 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:28.004235 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:28.004913 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:28.008550 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:28.039121 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:28.041749 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:28.041761 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.047228 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:28.053211 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:28.053912 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:28.093873 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:28.203864 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:01:28.232271 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:28.299211 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:28.300475 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:28.300899 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:28.307967 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:28.308794 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.323856 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:28.327550 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:28.370491 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:28.402932 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:28.404219 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:28.405398 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.413460 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:28.416556 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:28.422611 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:28.470909 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:28.485246 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.493374 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:28.496850 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:28.497366 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:28.499437 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.506381 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:28.508159 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:28.574647 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:28.574858 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.575945 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:28.585049 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:28.604832 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:28.606398 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:28.607286 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.661190 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:28.662510 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:28.664961 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.667018 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:28.667574 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:28.669596 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:28.680207 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:28.763183 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:28.764802 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:28.766212 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:28.769416 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:28.776131 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:28.777891 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.787444 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:28.787839 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:28.790280 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:01:28.888737 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:28.891294 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:28.891399 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:28.895658 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:28.904362 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.905164 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:28.917428 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:01:28.936975 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:28.943715 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:28.953936 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:28.955531 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.956918 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:28.966501 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:28.969188 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:29.058055 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:29.058782 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.061419 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:29.064199 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:29.064908 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.067756 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:29.087036 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:29.125943 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:29.219534 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.221086 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:29.223801 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:29.227963 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:29.236006 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:29.239470 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.240886 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:29.320643 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:29.372828 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:29.380113 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:29.383816 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:29.385303 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.388657 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.415997 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:29.424270 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:01:29.457532 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:29.474601 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:29.489029 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:29.495337 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.504956 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.505208 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:29.515034 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:29.530409 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:29.549844 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.550535 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:29.551975 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:29.561995 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.564733 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:29.573784 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:29.600921 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:29.602979 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.606467 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:29.608495 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:29.616818 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:29.617534 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:29.631239 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.659430 14217 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:00:14.146298 14223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:14.146545 14223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:14.150388 14223 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:00:15.218080 14223 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:00:15.685181 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.685300 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.685418 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.685465 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.685599 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.685662 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.685715 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.685763 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.685811 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.685856 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.685909 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.685976 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.686018 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.686066 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.686133 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.686167 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
I20240503 14:00:15.853178 14223 rebalancer_tool.cc:1074] tablet f5c06806a19e4f4592d26ed4acc651cd: '623e8dcb58544e558207d8e652dfed5f' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move scheduled
I20240503 14:00:15.864722 14223 rebalancer_tool.cc:1084] tablet c2e9f6eecbec45988b290775277dbddc: '8b17d706d2344dbaae13cda9b0733143' -> '1b4a61e8a11440e7a9f4b300d2442f75' move ignored: Illegal state: tablet c2e9f6eecbec45988b290775277dbddc: replica 1b4a61e8a11440e7a9f4b300d2442f75 is already present
I20240503 14:00:15.864825 14223 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:16.197242 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.203421 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.203510 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.203649 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.203693 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.203732 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.203766 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.203804 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.203863 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.203910 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.203986 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.204028 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.204061 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.204110 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.204155 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.204190 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.204217 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.204337 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.204396 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.204432 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.204500 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
I20240503 14:00:16.296026 14223 rebalancer_tool.cc:1074] tablet 3481715d755a4277b0905b0832fad4af: '8b17d706d2344dbaae13cda9b0733143' -> '66190085d1c04e068d5c0dec9c4bf07d' move scheduled
I20240503 14:00:16.477528 14223 rebalancer_tool.cc:1084] tablet a4b4e16598b740cb9d690c2442a3e994: '84c63c6f551949f6a1d7c07760824361' -> 'b67f337e10994234baedbf0b19c13513' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240503 14:00:16.477602 14223 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:17.165114 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.165220 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.165261 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.165334 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.165381 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.165418 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.165454 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.165491 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.165527 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.165561 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.165592 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.165629 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.165663 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.165699 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.165733 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.165769 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.165814 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.165856 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:00:17.178458 14223 rebalancer_tool.cc:1074] tablet f4f9cb87c70e43dbb0d850c275ebb8a5: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:17.263470 14223 rebalancer_tool.cc:1074] tablet 0e442bb664f44887aa8afdb62e1d97fc: '84c63c6f551949f6a1d7c07760824361' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move scheduled
I20240503 14:00:17.328517 14223 rebalancer_tool.cc:1074] tablet b031f09b97664d1a94c14f0661181899: '623e8dcb58544e558207d8e652dfed5f' -> '1b4a61e8a11440e7a9f4b300d2442f75' move scheduled
I20240503 14:00:17.450276 14223 rebalancer_tool.cc:1074] tablet 355b5456887b4c5694badc9d53b1809a: '623e8dcb58544e558207d8e652dfed5f' -> 'b67f337e10994234baedbf0b19c13513' move scheduled
I20240503 14:00:17.481969 14223 rebalancer_tool.cc:1084] tablet 4224ca4d93ce4784b52a7550468ce1b1: '84c63c6f551949f6a1d7c07760824361' -> '66190085d1c04e068d5c0dec9c4bf07d' move ignored: Illegal state: tablet 4224ca4d93ce4784b52a7550468ce1b1: replica 66190085d1c04e068d5c0dec9c4bf07d is already present
I20240503 14:00:17.482059 14223 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:17.972383 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.972476 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.972553 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.972604 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.972645 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.972712 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:18.037890 14223 rebalancer_tool.cc:1074] tablet c2e9f6eecbec45988b290775277dbddc: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:18.047935 14223 rebalancer_tool.cc:1074] tablet d837845c6f644bca83315c9f7c45246d: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:18.068219 14223 rebalancer_tool.cc:1084] tablet df13ec098dc548afac8c67195d25e7e8: '84c63c6f551949f6a1d7c07760824361' -> '1b4a61e8a11440e7a9f4b300d2442f75' move ignored: Illegal state: tablet df13ec098dc548afac8c67195d25e7e8: replica 1b4a61e8a11440e7a9f4b300d2442f75 is already present
I20240503 14:00:18.068292 14223 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:18.675851 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:18.675997 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:18.676085 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:18.676126 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:18.676183 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
I20240503 14:00:18.691082 14223 rebalancer_tool.cc:1074] tablet a0106f7a40cb467eb4738890ac4e22d4: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:18.729094 14223 rebalancer_tool.cc:1074] tablet faa0a6ca00d442eaaf9fca7fc1259ec7: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:21.535068 14223 rebalancer_tool.cc:1123] tablet 3481715d755a4277b0905b0832fad4af: '8b17d706d2344dbaae13cda9b0733143' -> '66190085d1c04e068d5c0dec9c4bf07d' move is abandoned: Not found: Server with UUID 8b17d706d2344dbaae13cda9b0733143 not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:21.669703 14223 rebalancer_tool.cc:1136] tablet 0e442bb664f44887aa8afdb62e1d97fc: '84c63c6f551949f6a1d7c07760824361' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move completed: OK
I20240503 14:00:21.739544 14223 rebalancer_tool.cc:1136] tablet d837845c6f644bca83315c9f7c45246d: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:22.301502 14223 rebalancer_tool.cc:1136] tablet f4f9cb87c70e43dbb0d850c275ebb8a5: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:23.664171 14223 rebalancer_tool.cc:1074] tablet 211de6f35a844e278e8c1eb1621e7c5f: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:23.673980 14223 rebalancer_tool.cc:1074] tablet 79b21b1e5292488aad442060b417f3c1: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:24.144946 14223 rebalancer_tool.cc:1136] tablet faa0a6ca00d442eaaf9fca7fc1259ec7: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
I20240503 14:00:24.168041 14223 rebalancer_tool.cc:1136] tablet c2e9f6eecbec45988b290775277dbddc: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
I20240503 14:00:24.187407 14223 rebalancer_tool.cc:1136] tablet a0106f7a40cb467eb4738890ac4e22d4: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:24.200973 14223 rebalancer_tool.cc:1136] tablet f5c06806a19e4f4592d26ed4acc651cd: '623e8dcb58544e558207d8e652dfed5f' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move completed: OK
I20240503 14:00:24.655750 14223 rebalancer_tool.cc:1136] tablet b031f09b97664d1a94c14f0661181899: '623e8dcb58544e558207d8e652dfed5f' -> '1b4a61e8a11440e7a9f4b300d2442f75' move completed: OK
I20240503 14:00:25.154865 14223 rebalancer_tool.cc:1074] tablet 70ffc5888a5945d6a8e166420e6f1bd9: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:25.188369 14223 rebalancer_tool.cc:1074] tablet 4224ca4d93ce4784b52a7550468ce1b1: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:25.210053 14223 rebalancer_tool.cc:1074] tablet a4b4e16598b740cb9d690c2442a3e994: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:25.240828 14223 rebalancer_tool.cc:1136] tablet a4b4e16598b740cb9d690c2442a3e994: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:25.308432 14223 rebalancer_tool.cc:1136] tablet 4224ca4d93ce4784b52a7550468ce1b1: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
I20240503 14:00:25.462518 14223 rebalancer_tool.cc:1136] tablet 70ffc5888a5945d6a8e166420e6f1bd9: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:25.522636 14223 rebalancer_tool.cc:1136] tablet 211de6f35a844e278e8c1eb1621e7c5f: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:25.551846 14223 rebalancer_tool.cc:1136] tablet 79b21b1e5292488aad442060b417f3c1: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:25.583854 14223 rebalancer_tool.cc:1136] tablet 355b5456887b4c5694badc9d53b1809a: '623e8dcb58544e558207d8e652dfed5f' -> 'b67f337e10994234baedbf0b19c13513' move completed: OK
I20240503 14:00:25.869529 14223 rebalancer_tool.cc:1074] tablet 2089eeda1e0f4ff5926584d472e6631d: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:25.886399 14223 rebalancer_tool.cc:1074] tablet 7d2e1517a8ac4041bfb4b90634cb0fa6: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:25.906090 14223 rebalancer_tool.cc:1074] tablet 92506cde7b084b14b88d4a915731eeb3: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:25.929069 14223 rebalancer_tool.cc:1074] tablet 6c4f3bc64e7145e2b692beab730d1496: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:25.970515 14223 rebalancer_tool.cc:1074] tablet 11a8c270532947ba9b505aace44b56f6: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:26.121305 14223 rebalancer_tool.cc:1136] tablet 11a8c270532947ba9b505aace44b56f6: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:26.480764 14223 rebalancer_tool.cc:1136] tablet 7d2e1517a8ac4041bfb4b90634cb0fa6: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:26.492096 14223 rebalancer_tool.cc:1136] tablet 92506cde7b084b14b88d4a915731eeb3: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:26.501190 14223 rebalancer_tool.cc:1136] tablet 2089eeda1e0f4ff5926584d472e6631d: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:26.510372 14223 rebalancer_tool.cc:1074] tablet c42fee9fadbb4601a7fe32eb6a04e767: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:26.531850 14223 rebalancer_tool.cc:1074] tablet e92d8f55591346158b9583feb958a8ca: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:26.553351 14223 rebalancer_tool.cc:1074] tablet 5e6ca408c4254678b67a74f49a34881b: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:26.668190 14223 rebalancer_tool.cc:1136] tablet 5e6ca408c4254678b67a74f49a34881b: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:26.693132 14223 rebalancer_tool.cc:1136] tablet e92d8f55591346158b9583feb958a8ca: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:27.313947 14223 rebalancer_tool.cc:1136] tablet 6c4f3bc64e7145e2b692beab730d1496: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:27.339197 14223 rebalancer_tool.cc:1136] tablet c42fee9fadbb4601a7fe32eb6a04e767: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
W20240503 14:00:27.753618 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 1b4a61e8a11440e7a9f4b300d2442f75 to server b67f337e10994234baedbf0b19c13513
I20240503 14:00:27.769161 14223 rebalancer_tool.cc:1084] tablet f4f9cb87c70e43dbb0d850c275ebb8a5: 'e85b0283a0fd4e3ba81b4170f3cdacb5' -> '66190085d1c04e068d5c0dec9c4bf07d' move ignored: Illegal state: tablet f4f9cb87c70e43dbb0d850c275ebb8a5: replica 66190085d1c04e068d5c0dec9c4bf07d is already present
I20240503 14:00:27.769235 14223 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:28.183892 14223 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server e85
W20240503 14:01:29.688798 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:29.719184 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:29.725051 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:29.725750 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:29.729192 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:29.733696 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.745415 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:29.746731 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:29.753938 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.761996 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:29.762056 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:29.765777 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:29.765933 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:29.766983 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:29.772559 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:29.782857 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:29.818228 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:29.878114 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:29.881500 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.882651 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:29.882995 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:29.884687 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:29.892443 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:29.934141 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:01:29.975754 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:01:30.003945 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:30.076097 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:30.078860 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.080013 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:30.080021 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:30.081838 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240503 14:01:30.082557 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:30.087414 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:30.088682 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:30.099395 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:30.131482 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:01:30.155221 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:01:30.179209 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:01:30.195087 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:30.195801 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:30.205833 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:30.207688 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.208768 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:30.212709 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:30.214645 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:30.219537 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:30.222517 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.229032 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.239670 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:30.254159 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:30.260280 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:30.271036 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.271129 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:30.338809 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:30.341260 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.343302 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.344879 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:30.352167 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:30.354730 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:30.456039 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:30.457093 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.458194 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.458591 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:30.466636 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:30.482810 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:30.516834 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:30.574342 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:30.575611 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:30.575937 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:30.581192 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:30.599633 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 14:01:30.600271 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:30.610625 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:30.617899 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:30.622555 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.622608 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:30.633073 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:30.641932 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:30.646145 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:30.647027 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:30.647450 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:30.648363 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:30.706820 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:30.758935 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:30.763342 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.769634 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:30.774091 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:30.776378 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:30.785797 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:30.791095 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:30.792071 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.798278 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.800793 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:30.805976 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:30.808977 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:30.817569 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:30.867175 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:01:30.873672 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:30.929524 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:31.067485 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 14:01:31.125944 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:31.126369 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.130620 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:31.130985 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.134009 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:31.149647 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:31.155452 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:31.168265 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.174612 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:31.175860 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:31.183975 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:31.186920 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:31.186939 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:31.197716 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:31.210942 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:01:31.273370 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:31.296718 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:31.303407 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.304201 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:31.306492 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.307571 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:31.316527 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:01:31.317392 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.321288 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:31.327587 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:31.328658 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:31.331646 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:31.351655 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:31.362607 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.453290 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:31.454159 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:31.454413 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:31.463727 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:31.467280 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:31.475014 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:31.483832 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:31.485745 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:31.565501 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:31.565687 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:31.574226 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:31.575811 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.586468 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:31.588164 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:31.594120 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:31.596117 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:01:31.632761 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:31.655143 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:31.683944 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:31.688966 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:31.692690 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:31.695330 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:31.701468 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.704731 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:31.713768 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:31.743333 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:31.744762 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:31.747736 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:31.750434 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:31.753614 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.756091 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.840970 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:31.841161 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.844805 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:31.847383 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:31.853842 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:31.870905 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:32.010376 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:32.024871 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:32.029812 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:32.050246 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:32.050894 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:32.051146 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.052313 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:32.053254 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:32.056250 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:32.089566 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:32.168112 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:32.179446 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:32.181723 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.183034 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.189853 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:32.190124 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:32.196406 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:32.196465 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.204113 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:32.205951 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:32.209432 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:32.233757 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:32.235715 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:32.236290 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:32.322659 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:32.340070 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.341691 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:32.344864 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.345913 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:32.346582 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:32.350026 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:32.364097 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:32.366464 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.367349 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:32.371649 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:32.385442 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:32.389811 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:32.411058 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:32.414211 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:32.419016 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:32.420936 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:32.491863 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.492537 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:32.495177 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:32.496376 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:32.496567 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:32.497102 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.497690 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:32.537988 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:01:32.552006 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:32.594012 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:32.596421 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:32.596813 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.599614 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:32.615805 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.628095 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:32.648558 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:32.698828 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:32.700450 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:32.704138 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:32.704237 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:32.704985 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:32.710186 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.715320 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:32.720108 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:32.720855 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:01:32.744858 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:32.813292 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:32.813865 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.820508 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:32.820885 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:32.820977 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:32.821204 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.915159 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:32.916460 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:32.916476 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.921736 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:32.921999 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:32.922816 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:32.936825 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:33.006268 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:01:33.015041 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:33.029536 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:33.029632 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:33.036644 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.036976 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:33.037369 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:33.039122 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.148941 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:33.150436 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:33.150846 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:33.151821 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:33.161034 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:33.167004 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:33.170750 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:33.171018 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.177354 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.182179 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:33.184288 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:33.191121 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:33.194465 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.226497 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:33.262681 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:01:33.276646 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:01:33.310704 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:33.322291 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:33.322430 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:33.324513 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.332790 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:33.338821 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.345726 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:33.366859 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:33.368695 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:33.373363 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:33.378002 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.390475 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.411327 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:33.428303 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:33.472148 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:33.491331 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:33.493563 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:33.496202 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.498971 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.510001 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:33.522094 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:33.596443 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:01:33.606897 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:33.663311 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:33.666163 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.667107 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:33.668612 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:33.678718 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:33.682579 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:33.755995 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:33.769704 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:33.782517 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.785996 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.786914 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:33.791404 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:33.797369 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:33.809584 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:33.851537 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:33.854277 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:33.854709 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:01:33.855129 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.864392 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:33.868319 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:33.868625 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:33.871040 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:33.939288 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:33.964215 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:33.966578 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:33.966841 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.978410 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:33.980618 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:33.991693 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:34.083000 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:34.084874 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:34.085518 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:34.088809 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.102684 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:34.104539 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:34.129413 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:34.131552 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:34.142367 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.145493 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:34.150774 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:34.154292 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:34.154850 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:34.193058 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:34.248208 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:34.253376 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:34.257463 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.267740 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:34.271520 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.280740 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:34.295046 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:34.296825 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.301486 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.304270 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:34.339946 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:34.340934 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:34.355214 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:34.367444 15019 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:01:34.367566 15019 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:01:34.368089 15019 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:01:34.368613 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:01:34.368721 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:01:34.368785 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:01:34.369400 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:34.369518 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:34.369587 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:01:34.369782 15019 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:01:34.420164 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:34.421789 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:34.422473 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.423467 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:34.425091 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:34.427975 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:34.432884 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:34.464329 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:34.473681 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:34.513622 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:34.560915 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:34.667726 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:34.669536 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:34.675469 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.676184 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:34.682276 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:34.695782 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:34.740128 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:34.775146 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:34.778381 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:34.783653 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:34.786598 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:34.794427 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:34.803287 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.811637 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:34.835685 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:34.839124 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:34.842473 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:34.843068 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.848945 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:34.861160 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:34.862377 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:34.867653 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.940290 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:34.942570 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:34.942886 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.945263 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:34.947420 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:34.953258 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:34.954219 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:35.006163 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:35.011778 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:35.042917 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:35.059657 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:35.066605 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:35.068256 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:35.068642 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.069015 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:35.069752 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:35.077926 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:35.124368 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:35.141855 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:35.158340 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 164 similar messages]
I20240503 14:01:35.181608 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:35.184000 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:35.192126 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:35.195144 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:35.202883 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:35.203924 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:35.204782 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.205248 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.207134 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:35.216271 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.232791 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:35.252444 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:35.259030 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:35.287508 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:35.335932 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:35.344102 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:35.345211 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:35.347393 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:35.357049 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.364192 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.393597 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:35.395627 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:35.397061 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:35.407752 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.410765 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:35.413218 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:35.421077 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.429077 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:35.453370 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:35.464267 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:35.509251 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:35.515398 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:35.551781 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:35.611316 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240503 14:01:35.706923 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:35.707216 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:35.708734 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:35.709650 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.722909 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:35.735952 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:35.752601 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.766865 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:35.767786 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:35.787325 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.789443 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.790246 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:35.792786 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:35.820335 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:01:35.877082 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:35.878999 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:35.879642 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.886339 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:35.904753 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:35.921223 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.951121 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:35.956808 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:35.961719 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:35.967175 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.970245 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:35.975780 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:35.976557 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:35.997241 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:36.058305 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:36.061331 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.063371 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:36.065049 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:36.065337 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:36.073829 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:36.081585 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:36.088594 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:01:36.134088 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:36.163868 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:36.167589 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.169170 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:36.173004 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:36.173260 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:36.185655 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:36.193656 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.255357 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:36.259598 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:36.261092 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.261785 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:36.262612 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.269822 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:36.330314 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:36.330538 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:36.335253 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:36.337898 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.340775 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:36.341365 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.411340 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:36.415416 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:36.415840 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:36.418584 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:36.421298 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.429441 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:36.469113 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:36.541203 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:01:36.550724 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:36.602607 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:36.604997 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:01:36.609308 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:36.625021 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:36.665767 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:36.666466 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:36.667420 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:36.669477 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:36.673969 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.674876 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:36.693950 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:36.745822 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:36.748781 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:36.751833 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.763549 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.777297 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:36.777920 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:36.837554 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:36.838243 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:36.838383 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:36.850207 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.850819 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:36.860128 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:36.912426 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:36.917816 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:36.921125 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.923410 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:36.928129 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:36.948388 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:36.949885 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:37.006826 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:37.006927 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:37.007587 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.009435 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:37.009457 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:37.011001 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:37.012382 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:37.024418 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:37.106999 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.107708 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:37.107980 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:37.115499 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:37.121546 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.127856 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:37.151748 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:37.151903 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:37.197646 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:37.208168 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:37.224006 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:37.229108 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.231616 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:37.233856 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:37.238123 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:37.239223 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.273801 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:37.276202 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.279433 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:37.282258 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:37.283538 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:37.287429 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:37.289819 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:37.324460 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:37.380573 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:37.393862 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:37.396180 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.397706 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:37.397938 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:37.400709 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.407431 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:37.416556 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:37.474606 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:37.528962 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:37.581452 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:37.586062 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:37.586251 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.587556 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:37.589598 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:37.601704 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:37.608187 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:37.609827 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:37.619786 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:37.620302 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:37.630215 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:37.709273 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:37.718163 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:37.718238 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:37.725059 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.729748 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.732007 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:37.745640 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:37.758232 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.758222 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:37.759708 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.772291 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:37.777161 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:37.778147 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:37.801863 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:37.879426 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:37.881260 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:37.884641 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.888316 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.890640 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:37.895655 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:37.898196 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:37.912762 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:37.925025 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:37.935994 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:37.939392 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:37.957203 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:37.968084 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:37.969691 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:38.004884 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:01:38.042012 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:38.048450 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:38.051901 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:38.053015 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.057443 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:38.062314 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:38.062659 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.071406 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:38.074213 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:38.154784 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:38.155447 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.156196 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.156924 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:38.180353 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:38.188321 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:38.207489 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:38.222178 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:38.245536 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:38.245779 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.246753 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:38.257472 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.262617 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:38.270737 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:38.323751 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:38.363170 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.363199 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:38.370239 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:38.379516 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:38.379877 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:38.381147 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:38.386353 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.477429 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:38.478753 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.479321 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.484069 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:38.485359 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:38.500082 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:38.583137 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:38.584264 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:38.588006 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:38.592532 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.595474 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.600003 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:38.607489 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:38.617210 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:38.697168 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:38.697654 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.702247 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:38.718791 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:38.718927 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:38.722718 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.749531 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:38.751600 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:38.754240 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:38.756853 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:38.767939 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.778276 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:38.824831 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:38.879563 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:38.883927 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.887584 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:38.887822 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:38.888279 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.896389 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:38.899540 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:38.925626 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:38.926111 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:38.927071 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:38.928290 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:38.930650 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:38.941632 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:01:38.949362 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:38.952193 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.043325 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:39.045400 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:39.054404 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:39.054453 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.054718 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:39.055123 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:39.062618 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:39.087131 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:01:39.159794 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:39.212738 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.216089 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:39.219781 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.222229 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:39.223104 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:39.229208 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:39.318662 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:39.322355 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:39.322655 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.330416 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.338716 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:39.341351 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:39.348870 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:39.363715 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:39.402628 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:39.405925 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:39.410089 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:39.410089 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:39.424046 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:39.425786 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:39.426777 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.502805 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:39.505225 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.507375 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:39.507644 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:39.512125 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.522128 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:39.536080 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:39.579596 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:39.612361 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:39.615028 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:39.621555 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:39.633796 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:39.640215 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.640517 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:39.642697 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:39.693847 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:39.698086 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:39.700174 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:39.703177 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:39.703584 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.704864 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.711120 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:39.739992 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:39.786423 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:39.787600 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:39.788646 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.794453 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:39.796800 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:39.802559 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:39.803118 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:39.805490 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:39.872515 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.872712 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:39.875193 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:39.876363 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:39.884079 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:39.887600 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.894111 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:39.919653 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:39.940644 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:39.944118 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:39.945515 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:39.949154 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:39.950373 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:39.960865 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:39.977453 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:01:39.993232 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:01:40.046062 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:01:40.091704 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:40.175303 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:40.188580 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:40.190492 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:40.190825 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.192322 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:40.194084 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:40.194963 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.203426 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:40.212307 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 157 similar messages]
I20240503 14:01:40.283906 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:40.292501 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.296651 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.298223 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:40.304428 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:40.304426 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:40.310485 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:40.335134 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.345944 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.346169 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:40.350056 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:40.370038 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:40.372215 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:40.372283 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:40.395140 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:40.452061 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:40.453518 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:40.455543 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:40.459344 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.464789 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.473887 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:40.558652 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:40.559485 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:40.560526 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:40.561813 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:40.562516 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:40.571529 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:40.580765 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:40.623996 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 14:01:40.624518 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:01:40.628872 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:01:40.631933 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:40.663594 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:40.677016 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:40.684258 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.688633 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.692818 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:40.696452 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:40.697070 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:40.697155 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:40.701267 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:40.713007 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:40.720265 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.731331 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:40.738677 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:40.741420 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:40.747375 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.862426 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:40.866701 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:40.870123 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:40.870708 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:40.880584 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:40.883402 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.884510 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.886332 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.887630 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:40.888260 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:40.888878 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:40.909988 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:40.978405 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:40.979842 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:01:41.069067 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:01:41.069238 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:41.138623 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:01:41.181630 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:41.225680 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:41.229023 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:41.230798 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.232851 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:41.241896 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.247916 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:41.254498 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:41.263415 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:41.282871 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:41.284391 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.285908 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.286514 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:41.293848 17985 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:01:41.293947 17985 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:01:41.294281 17985 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:01:41.295765 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:41.295871 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:41.295910 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:01:41.296072 17985 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:01:41.358413 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:41.388902 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.390790 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:41.398556 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:41.398890 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:41.399633 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:41.412206 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.440158 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:41.444204 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:41.446960 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.457054 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.458755 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:41.459558 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:01:41.465035 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:41.507648 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:01:41.546936 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:41.547410 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:41.551118 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.551623 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:41.553601 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:41.563162 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.565403 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:41.605428 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:41.660650 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:41.662389 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:41.662918 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:41.668493 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.671517 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:41.677744 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:41.688638 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:41.692516 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:41.703166 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 14:01:41.752545 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:41.755163 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:41.759083 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.762277 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.770087 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:41.784230 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:41.835338 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:41.836547 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:41.843422 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:41.844866 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:41.851220 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:41.860774 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:41.867241 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.908413 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:41.911181 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:41.911674 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:41.913784 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:41.927397 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:41.932574 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:41.990649 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:01:42.067898 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:42.086941 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:01:42.136238 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:42.150045 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:42.150724 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:42.155171 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.157763 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:42.167377 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:42.173166 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:42.257778 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:42.265106 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:42.270504 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.272071 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:42.278263 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:42.285948 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:42.290551 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:42.291755 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.291996 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:42.295192 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:42.301937 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:42.324965 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:01:42.335481 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:42.341077 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.346598 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:42.375471 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:01:42.426174 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:42.448452 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:42.453954 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:42.457381 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:42.461028 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:42.461138 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:42.461171 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:42.472715 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:42.475507 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:42.479703 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:42.489764 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:42.489759 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:42.496589 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:42.497916 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:42.501914 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.511893 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:42.619908 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:42.622270 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.622500 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:42.624136 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:42.625036 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:42.630052 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:42.632949 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:42.640317 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:01:42.648646 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:42.666155 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:42.719549 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:42.728170 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:42.729924 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:42.737370 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:42.744036 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:42.758111 19308 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:01:42.758231 19308 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:01:42.758685 19308 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
W20240503 14:01:42.759821 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:42.762979 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
W20240503 14:01:42.763053 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:42.763099 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
W20240503 14:01:42.763125 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:42.763161 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
I20240503 14:01:42.763167 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:01:42.763419 19308 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:01:42.765249 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.777098 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.842080 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.846123 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:42.846441 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:42.848155 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.856124 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:42.870572 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:42.946496 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:42.963908 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:42.966631 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:42.970468 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.975656 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:42.976040 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:42.990301 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:43.041491 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:43.060992 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:43.069502 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:43.069770 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:43.072544 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.079684 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.080224 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:43.084090 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:43.084189 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:43.160552 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:01:43.171422 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:43.180749 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:43.182067 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:43.184868 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.194263 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:43.196980 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.207541 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:43.220788 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:01:43.233585 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:43.288318 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:43.289928 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:43.292404 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.300987 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:43.312709 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.318771 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:43.319065 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:43.325260 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:43.348628 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:43.349609 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.351832 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:43.368525 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.369318 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:43.424731 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:43.474251 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.478344 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:43.478701 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.479205 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:43.479205 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:43.495282 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:43.495517 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.499279 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:43.516827 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:43.518244 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:43.535811 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:43.536579 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.546273 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:43.561653 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:43.633908 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.638824 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:43.648252 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:43.649832 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:43.653096 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:43.663434 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:43.667306 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:43.684371 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:01:43.730803 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:43.793273 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:43.832953 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:43.837754 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.839072 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:43.839490 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:43.842255 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:43.855279 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.957243 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:43.973735 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:43.974049 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:43.975342 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:43.976547 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:43.987386 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:43.996497 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:43.996861 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.000448 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:44.003443 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:44.019804 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.030853 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:44.042507 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:44.120514 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:44.124933 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:44.125034 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.125383 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:44.129813 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.143813 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:44.171813 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:44.178301 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:44.231745 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:44.234182 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.237052 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:44.238617 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:44.242559 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:44.266508 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:01:44.267517 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.339466 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:44.339901 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:44.341449 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.345119 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:44.348922 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:44.385907 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:44.392688 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:44.404296 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:44.414547 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:44.417114 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:44.420538 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:44.423996 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:44.434513 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.446200 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:44.542852 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:44.544400 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:44.547647 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:44.548379 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:44.549789 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.554795 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:44.561379 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.563102 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:44.570006 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:44.606101 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:44.606619 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.607055 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.611033 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:44.612824 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:44.615259 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:44.655980 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:44.723286 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:01:44.817098 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:01:44.824129 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:01:44.858273 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:44.871227 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.874123 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:44.874753 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:44.876827 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:44.878358 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:44.901907 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:44.908149 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:01:44.953732 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:44.971035 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:44.974655 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:44.976562 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:44.981554 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:44.992926 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:44.992983 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:44.993141 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:44.995755 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:45.035643 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.038725 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:45.040787 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:45.048188 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:45.048388 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:45.049329 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:45.058508 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:45.062444 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:45.134508 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.136458 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.136493 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:45.140136 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:45.152576 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:45.160382 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:45.200481 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:01:45.242746 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:45.243089 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:45.243837 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.245002 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:45.248322 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:45.259603 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.262137 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:45.286609 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240503 14:01:45.304419 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:45.362383 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:45.373222 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:45.376955 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.387990 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.389329 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:45.391474 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:45.391579 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.391878 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:45.399804 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:45.408350 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:45.411901 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:45.412415 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:45.441116 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:45.479493 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:01:45.535820 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:45.541512 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:45.543962 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:45.552239 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:45.553987 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.557082 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:45.558555 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.561384 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:45.562371 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:45.563398 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:45.565232 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:45.578876 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:45.581133 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:45.588973 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:45.599637 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:45.624049 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:45.641835 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:01:45.676893 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 14:01:45.677484 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:01:45.847990 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:45.900399 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:45.909958 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:45.913560 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:45.916589 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.924182 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.932157 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:45.943332 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:45.957664 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:45.964545 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:45.966974 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:45.971431 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:45.977932 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:45.990301 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:45.991874 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.086709 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:46.090003 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.090241 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:46.104961 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:46.107661 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:46.125000 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:46.126322 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.126435 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:46.129084 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.132838 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.141534 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:46.174199 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:46.177664 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:46.188496 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:46.258636 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:46.259629 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:46.260217 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:46.262183 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 14:01:46.263017 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:46.263670 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.278254 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:46.302816 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:01:46.333019 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:46.361995 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:46.364028 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:46.365731 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.367210 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:46.385342 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:46.386440 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.386826 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:46.409786 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:46.484546 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.490289 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:46.499922 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:46.500618 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.503602 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:46.506392 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:46.529553 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:46.539891 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.540547 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:46.541214 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.549507 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:46.556730 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:46.618134 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:46.648223 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:46.653301 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:46.654942 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:46.655416 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.657466 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:46.660120 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:46.802621 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:01:46.848095 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:01:46.853797 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:46.855127 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:46.855896 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:01:46.861270 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:46.869158 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:46.871403 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:46.874092 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:46.898623 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:46.914204 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:46.980129 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:46.981256 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:46.986968 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:46.989200 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:47.001863 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:47.001863 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.017107 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:47.018126 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.023134 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.028024 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:47.028857 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:47.039932 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:47.043663 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:47.045768 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:47.067543 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:47.102037 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:47.158697 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:47.161201 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.165899 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:47.165935 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:47.170562 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.175416 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:47.195080 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:47.201508 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.210325 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:47.214120 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.222651 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:47.224617 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:47.317687 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:47.319186 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:47.323063 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:47.323309 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.323348 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:47.329123 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:47.364638 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:01:47.385141 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:01:47.409762 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:01:47.430176 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:47.436131 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:47.436309 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.438177 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:47.438751 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.439199 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:47.456666 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:47.457590 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:47.504223 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:47.523636 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:47.527580 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:47.528437 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.533133 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:47.533450 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:47.536991 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:47.538938 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:47.543870 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:47.568440 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:47.639798 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:47.641559 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.642117 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:47.651638 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.658151 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:47.661691 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:47.670711 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:47.692314 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:47.754371 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:47.754987 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:47.759384 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.760869 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.762907 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:47.774260 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:47.783821 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:47.878893 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:47.879034 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.879278 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:47.879714 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:47.880080 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:47.892140 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:47.906524 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:47.935195 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:47.956209 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:47.984862 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:47.993604 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:47.993814 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:47.993961 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:47.996040 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.011998 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:48.018949 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:01:48.026237 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:48.040421 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:48.054137 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:48.056560 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.059949 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:48.060770 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.070036 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:48.078547 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:48.097685 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:48.165591 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:48.174202 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.178753 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:48.180467 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.189159 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:48.197496 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:48.198712 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:01:48.224339 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:48.227803 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:48.228439 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:48.228668 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:48.233096 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:48.236081 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:48.238464 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:48.239784 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:48.290999 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:48.337221 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:48.338256 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:48.341243 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.342689 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:48.348912 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:48.376390 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:48.388921 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:48.460258 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:48.536339 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.537205 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:48.538122 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:48.538300 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:01:48.544493 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:48.555490 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.569787 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:48.570534 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:01:48.616528 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:48.676146 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:48.678759 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.685153 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:48.685424 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.693297 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.700371 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:48.700692 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:48.703352 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.722815 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:48.725500 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:48.742533 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:48.742694 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:48.829219 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:48.830976 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:48.836417 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:48.843891 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:48.845166 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:48.854518 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.855041 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:48.933225 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:48.940680 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:48.944209 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:48.945111 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.948621 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:48.956104 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:48.968999 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:49.047717 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:49.049602 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:49.057612 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:49.064771 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:49.065061 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.067118 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:49.068675 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:49.109272 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:49.114046 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:49.121400 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.124930 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.137636 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:49.137928 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:49.157660 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:49.207808 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:49.209535 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:49.210441 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:49.212436 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.216672 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:49.219393 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.282734 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:49.286415 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.287338 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:49.288919 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:49.292171 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:49.299423 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:49.332525 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:49.405254 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:01:49.461392 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:01:49.504719 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:49.520722 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:49.523975 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:49.527745 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.528045 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:49.534788 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:49.535746 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:49.550873 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:49.562753 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:01:49.599503 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:49.637727 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:49.638512 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.640184 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:49.641595 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:49.644733 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:49.652139 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:49.652282 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:49.656644 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:49.706099 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:49.709336 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:49.718191 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.721405 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:49.721450 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.724251 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:49.793987 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:49.797919 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.803381 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:49.804864 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:49.806885 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:49.817977 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:49.821467 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:49.845775 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:49.866276 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:01:49.899528 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:49.904443 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:49.906318 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:49.908013 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:49.915441 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:49.916023 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:49.927573 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:49.951948 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:50.034201 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.044307 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:50.046767 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:50.054148 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:50.055611 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.057155 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:50.068012 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:50.070767 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:50.081310 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.083545 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.087268 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:50.097270 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:50.097703 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:50.099809 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:50.118250 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:50.166527 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:01:50.209003 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:50.218729 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:50.221426 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.222381 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.223551 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:50.231405 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:50.231645 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:50.233089 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:50.233597 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:50.239655 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:50.255959 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:50.268256 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.280166 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:50.287842 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:50.301530 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240503 14:01:50.304244 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:01:50.335676 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:01:50.364459 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:01:50.460089 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:50.579741 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:50.585093 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.591631 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:50.598685 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:50.606415 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:50.613188 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:50.622716 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.637892 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:50.640795 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:50.644029 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.645838 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:50.648021 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:50.662106 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:50.678916 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:50.697548 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 14:01:50.698261 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:01:50.699800 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:50.761497 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:50.767140 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:50.767225 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.774703 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:50.775661 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:50.780902 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:50.788152 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:01:50.825045 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:50.829499 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:50.830338 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:50.835047 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.835690 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.836131 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:50.855047 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:50.887645 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:50.910512 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:50.914029 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.914976 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:50.915692 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:50.916826 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:50.924680 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:51.025710 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:51.030623 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.032510 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:51.034461 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:51.034751 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.036753 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:51.040046 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:51.109645 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:51.139003 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:51.140965 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:51.144910 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:51.148471 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:51.149129 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.150327 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:51.154282 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:51.171554 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.203367 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:51.208576 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:51.208637 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.209661 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.214903 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:51.222772 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:51.225076 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:51.305850 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:51.309520 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:51.312593 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.314795 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:51.323683 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:51.323719 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:51.348961 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:01:51.414589 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:51.470238 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.471302 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:51.476068 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:51.479902 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:51.480999 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.492009 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:51.531545 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:51.581043 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:51.584311 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:51.586059 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:51.590170 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:51.593477 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.613065 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:51.635536 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:51.670040 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:51.674111 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.674273 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.674787 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:51.677249 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:51.687204 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:51.688575 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:51.716084 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:51.760423 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:51.764724 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:51.768288 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:51.768487 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.770951 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.783468 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:51.805594 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:51.834813 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:51.837575 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.837962 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:51.839704 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.844991 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:51.846637 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:51.894758 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:01:51.906186 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:01:51.918431 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:51.932443 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:51.936291 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:51.936502 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:51.938135 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:51.944347 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:51.967974 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:51.989166 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:01:52.047931 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:52.048686 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:52.054150 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.055423 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.064864 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:52.069276 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:52.073120 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:01:52.111294 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:52.136207 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:52.147709 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:52.150271 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:52.152652 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:52.152958 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.153281 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.176641 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:52.211513 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:52.227254 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:01:52.255648 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:52.258740 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:52.259471 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:52.259761 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:52.260396 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.269644 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:52.275702 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:52.303575 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:52.355842 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.356488 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:52.364197 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:52.367377 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:52.370945 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.379992 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:52.414264 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:01:52.457743 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:52.476696 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.477516 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:52.479629 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:52.483413 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:52.493310 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:52.493594 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:52.508668 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:52.582904 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:52.585759 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:52.587779 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:52.597724 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:52.602237 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:52.603976 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:52.607656 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.650941 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:52.654273 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.660508 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:52.663709 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:52.663928 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:52.672528 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:52.704988 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:52.705000 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:52.742935 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:01:52.750699 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:52.767175 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:52.769454 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:52.769757 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:52.772205 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.787209 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:52.787633 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.799736 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:52.837026 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:52.839486 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:52.839640 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:52.841439 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.849309 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.861027 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:52.905248 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:52.942473 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:52.950389 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:52.952689 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:52.953642 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:52.954895 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:52.955502 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:52.973807 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:53.092078 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:53.092855 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:53.094352 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:53.097461 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.103366 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:53.103515 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.112299 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:53.124639 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:01:53.142194 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:01:53.147802 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:53.181950 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:53.198246 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:53.202947 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:53.203531 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.205664 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:53.209430 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:53.223419 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.300505 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:53.301038 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:53.304988 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:53.308229 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:53.310624 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:53.312100 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:53.312152 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:53.326912 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:53.334009 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:53.388547 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:53.417989 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:53.419548 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:53.420300 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.421922 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:53.426930 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.440165 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:53.498903 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:01:53.506409 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:53.536216 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:53.545274 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:53.550555 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.552580 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:53.559465 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:53.564077 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.580302 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:53.588408 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:53.589190 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.599120 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:53.601905 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.604142 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:53.629596 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:53.725973 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:53.730259 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:53.735718 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:53.735999 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:53.742779 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:53.745555 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.751742 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:53.757771 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.761778 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:53.769902 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:53.785846 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.786756 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:53.831144 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:53.880138 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.881172 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:53.881920 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:53.884045 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:53.887645 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:53.892191 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:53.905515 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:54.030560 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:54.084302 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.085455 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:54.086241 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:54.093727 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:54.093972 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:54.104954 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:54.105372 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:54.116791 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:01:54.201547 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:54.206878 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.208531 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:54.215337 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:54.216178 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:54.224200 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:54.224810 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.226034 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:54.228785 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:54.229462 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:54.235386 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:54.235579 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:54.243779 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:54.254587 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.265448 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:54.318833 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:54.355513 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.357507 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:54.358810 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.360920 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:54.360975 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:54.375411 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:54.479629 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:54.485080 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:54.486601 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.495887 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:54.497733 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:54.500416 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:54.513664 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:54.515703 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:01:54.523548 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:54.585692 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:54.595917 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:54.599848 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:54.607674 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.610313 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:54.612394 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.629330 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:54.666430 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:54.667750 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:54.668258 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.682168 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:54.692597 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.696096 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:54.715406 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:54.763536 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:54.766685 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:54.770499 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.772043 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:54.772591 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.775913 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:54.802727 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:54.837548 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:54.843495 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.844743 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:54.844965 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:54.845633 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:54.849035 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:54.868242 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:54.934434 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:54.937497 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:54.943926 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:54.973524 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:55.030028 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:55.091022 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:55.092056 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.092541 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:55.092790 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:55.094758 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.099112 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:55.146265 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:55.157325 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:55.181373 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:55.188916 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:55.192013 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:55.192186 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:55.194820 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.202049 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:55.220238 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:55.251291 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:55.253208 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.253746 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:55.256062 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.259447 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:55.267657 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:55.317186 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240503 14:01:55.318287 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:55.348825 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:55.349005 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.349138 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:55.360314 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:55.382076 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:55.382292 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:55.414461 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:01:55.466939 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:55.467307 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.473112 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:55.479826 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:55.486171 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.487597 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:55.518342 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:55.556757 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:55.561745 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:55.583850 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:55.586905 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.588555 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:55.594534 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:55.610806 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.612928 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:55.615010 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.618767 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.621260 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:55.625429 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:55.663986 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:55.665149 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:55.668517 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:55.671813 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:55.731065 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:01:55.731812 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:55.736819 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:55.754015 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:55.754159 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.754523 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:55.754657 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:55.764014 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:55.766834 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:55.768332 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:55.810554 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:55.812960 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.815150 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:55.815552 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:55.818555 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:55.824538 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:55.856031 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:55.873848 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:55.966634 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:55.979789 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:56.073763 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:56.076084 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.079512 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:56.080711 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:56.086951 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:01:56.087409 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:56.099658 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:56.163720 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.164523 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:56.166966 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:56.168779 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:56.170168 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:56.173336 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:56.196038 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:56.240700 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:56.247027 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:56.249696 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:56.251744 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:56.257313 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.263815 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.323829 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:56.329278 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.329816 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:56.330876 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:56.331879 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:56.342290 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:56.344050 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:56.431360 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:56.435278 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:56.438452 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.440224 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:56.451396 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:56.458019 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:01:56.468222 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:56.471396 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:56.561549 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:56.563720 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:56.568300 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.574913 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:56.579291 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:56.583206 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.583384 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:56.589025 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.600838 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:56.603379 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:56.611327 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:56.611577 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:56.640375 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:56.655148 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:56.740985 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.744388 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:56.745015 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:56.749917 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:56.752066 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:56.755671 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.762075 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:56.767928 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:56.772092 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:56.784236 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:56.792201 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:56.792472 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:56.792505 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:56.804334 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:56.824149 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:56.830091 22772 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:01:56.830224 22772 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:01:56.830691 22772 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:01:56.831537 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:56.831743 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:56.831816 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:01:56.831934 22772 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:01:56.871354 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:56.872334 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:56.879222 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:56.988375 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:01:57.014541 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:01:57.091095 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:57.091859 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.093647 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:57.094789 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:57.095749 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:57.107615 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.147948 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.148344 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:57.153025 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:57.162940 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:57.166522 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:57.167531 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:57.184917 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:57.238824 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:57.252966 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:57.253576 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:57.261497 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.266258 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.277931 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:57.278223 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:57.296285 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:57.311540 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:57.312376 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:57.322741 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:57.324765 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.328056 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.341449 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:57.341737 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:57.364701 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:57.369176 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:57.371810 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:57.414964 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:57.418295 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.418424 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:57.419651 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:57.419763 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.444496 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:57.477962 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:57.522028 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:57.523653 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:57.528080 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.529330 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.538504 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:57.551419 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:57.634387 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:57.646010 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:57.646368 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.649447 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:57.658779 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.658974 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:57.659512 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:57.661178 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:57.692400 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:57.714838 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:57.717288 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:57.719682 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.721799 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:57.722641 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:57.729692 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:57.745641 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:57.751446 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:57.788772 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:57.808851 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:57.813294 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:57.828642 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.830045 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:57.833086 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.837613 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:57.843389 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:57.849864 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:57.977332 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:57.979429 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.980731 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:57.982553 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:57.984918 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:57.995326 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:01:58.062552 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:58.079866 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:58.087203 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.087397 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:58.087476 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:58.088874 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:58.099901 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.102712 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:58.121270 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:58.133702 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:58.186913 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:58.192773 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:58.192895 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.193289 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:58.193611 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:58.204430 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:58.246796 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:58.247826 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:58.249620 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:58.256791 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:58.264258 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:58.264772 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:58.277510 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:58.288498 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.328306 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:58.333155 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:58.334870 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:58.340370 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:58.340879 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.344422 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:58.352094 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:58.364987 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:58.384229 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:58.422044 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:58.425694 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:58.426501 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:58.427032 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:58.429352 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:58.430545 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.431433 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.529832 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.531463 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:58.540398 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:58.553647 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:58.554874 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.555552 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:58.573791 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:01:58.656960 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:58.660069 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.660497 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.662701 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:58.666368 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:58.686254 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:58.779302 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:58.783421 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:58.783628 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:58.788384 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.800705 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.809935 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:58.819504 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:58.833528 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:58.868513 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:58.881749 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.883461 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.887120 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:58.888200 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:58.892509 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:58.893947 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:58.968948 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:58.991984 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:58.992762 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:58.994096 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:58.999425 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.001406 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:59.007798 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:01:59.012131 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:59.100664 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:59.103102 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:59.107152 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.112277 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.129791 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:59.130816 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:59.137822 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.145258 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:59.146874 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:59.151199 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:59.154121 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:01:59.163456 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:59.170635 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.275406 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:59.276163 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:59.278820 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:59.286504 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.290789 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:59.292336 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:59.298454 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:59.300273 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:59.309624 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:01:59.315454 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:59.317255 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:59.320745 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:01:59.335888 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:59.337625 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.341332 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:59.416499 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:59.434302 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:59.434681 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:59.435848 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:59.436981 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.443809 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.446789 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:01:59.447094 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:59.486891 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:59.525230 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:01:59.542693 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:01:59.639644 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:59.643401 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.646113 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.646651 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:59.647984 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:59.651347 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:01:59.695693 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:01:59.743314 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:01:59.755474 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.757122 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.757174 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:59.759492 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:59.767390 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:59.774076 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:59.807544 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:01:59.815155 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:59.815470 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:59.817633 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:59.818126 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.829952 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:01:59.843259 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:01:59.858467 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:01:59.865644 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:01:59.915401 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:01:59.918781 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.921976 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:01:59.922331 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:01:59.932456 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:01:59.934540 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:01:59.977427 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:02:00.017087 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:02:00.018808 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:02:00.037140 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:00.038559 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:00.042126 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:00.049049 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.053997 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.054090 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:00.116055 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:02:00.152796 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:00.154017 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.155797 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:00.165612 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:00.167986 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:00.179114 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:02:00.191192 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:00.192751 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.195714 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:00.202822 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:00.203423 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.214427 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:00.225929 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:00.233986 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:00.262583 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:02:00.308471 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:02:00.321343 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 159 similar messages]
I20240503 14:02:00.326908 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:00.327580 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:00.328608 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.338594 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:00.345202 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:00.347788 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.348567 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:00.355998 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:00.356238 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:00.382565 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:00.390092 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.390360 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:00.392704 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:00.454144 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:00.499819 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:02:00.538825 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:02:00.649144 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:00.661233 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:02:00.679030 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.682190 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:00.683166 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:00.683212 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.691897 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:00.710417 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:02:00.714913 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:00.758776 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:02:00.760093 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.762843 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.763190 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:00.771456 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:00.780179 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:00.780499 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:00.829607 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:02:00.839843 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:02:00.840482 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:02:00.856576 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:00.876196 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:00.880097 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:00.882187 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.882516 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:00.896692 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.897624 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:00.937433 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:00.940786 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:00.941864 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.953604 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:00.964084 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:00.977150 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:01.049484 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:01.062944 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:01.064342 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.065037 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:01.067560 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:01.077420 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:01.082609 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:01.084964 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:01.113703 19308 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:02:01.113852 19308 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:02:01.114387 19308 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:02:01.114957 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:02:01.115096 12036 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:02:01.115185 12036 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:02:01.116127 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:01.116456 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:01.116546 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:02:01.116721 19308 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:02:01.180186 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:01.184453 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:01.194963 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.198496 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:01.202245 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.219424 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:01.260466 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:01.262847 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.267051 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.269971 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:01.272836 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:01.291983 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:01.296196 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:02:01.320959 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:02:01.359467 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.359916 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:01.363368 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:01.371397 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:01.385633 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.387372 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:01.402217 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:01.449483 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:01.452759 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:01.454139 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:01.455464 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.457350 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:01.458990 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:01.464432 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:01.542829 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:01.561704 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:01.681432 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:01.684484 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.684712 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:01.692195 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:01.697136 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.708562 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:01.739380 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:02:01.748598 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:01.817235 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:01.821839 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.821885 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:01.824129 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.833591 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:01.847419 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:01.847631 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.857954 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:01.867851 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:01.871377 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:01.878203 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:01.882797 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:01.898846 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:01.955595 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:01.960880 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:01.963318 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:02.009248 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:02.011310 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:02.014459 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:02.019115 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:02.025583 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:02.026070 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:02.028203 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:02.035065 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.035503 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.038594 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:02.041800 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:02.047571 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:02.055207 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.070989 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:02.129279 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:02:02.165113 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:02:02.166213 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:02:02.175071 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:02:02.208367 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:02.209362 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:02.212347 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.224165 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:02.229344 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:02.237712 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.349756 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.357237 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:02.357748 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:02.361717 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:02.363907 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:02.364307 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.443593 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:02.446790 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:02.449863 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:02.451807 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:02.460671 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:02.469914 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.472870 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:02.480773 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:02:02.486414 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:02.534603 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:02.543210 11884 maintenance_manager.cc:419] P e301624eacf4431890d4c35d6eed9916: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033386
I20240503 14:02:02.569068 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:02.571723 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.574086 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:02.575099 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:02.576022 11883 maintenance_manager.cc:643] P e301624eacf4431890d4c35d6eed9916: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.032s user 0.012s sys 0.011s Metrics: {"bytes_written":20879,"cfile_init":1,"dirs.queue_time_us":11007,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1115,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1017,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":329}
I20240503 14:02:02.577996 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:02.583752 11884 maintenance_manager.cc:419] P e301624eacf4431890d4c35d6eed9916: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7533 bytes on disk
I20240503 14:02:02.592445 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.595992 11883 maintenance_manager.cc:643] P e301624eacf4431890d4c35d6eed9916: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240503 14:02:02.631901 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:02.667845 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:02:02.672514 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:02:02.685452 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:02.689599 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.689888 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:02.690357 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.692147 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:02.692435 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:02.699501 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:02.785934 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:02.802788 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:02.808547 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:02.812803 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:02.819875 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.824293 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:02.825656 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:02.835427 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:02:02.849092 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:02:02.910887 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:02.922693 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:02.924952 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:02.931573 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:02.937759 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:02.943943 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:02.963685 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:02.964780 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:02.988015 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:02.989300 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:03.000133 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.000123 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:03.000458 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.013413 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:03.015034 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:03.071667 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:02:03.111147 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:03.119065 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:03.125797 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:03.130893 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:03.135239 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:03.135974 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.138608 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.157286 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:03.159157 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.171630 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:03.181681 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:03.197600 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:03.198346 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.200531 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:03.244922 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:02:03.283147 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:03.294131 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.295928 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:03.296880 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:03.297813 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.308180 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:03.309901 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:03.315193 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:02:03.335649 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:02:03.389449 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:03.470268 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:03.471515 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.471757 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:03.472157 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:03.482388 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.492012 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:03.590345 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:03.595455 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:03.596881 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:03.599345 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:03.607405 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.607678 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.622608 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:03.685076 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:03.685803 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.686230 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.688580 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:03.689201 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:03.700094 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:03.745747 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:03.791682 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:03.796352 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:03.796566 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:03.802517 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:03.807971 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:03.810241 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:03.827726 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:03.829756 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:03.838644 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:03.925443 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:03.929479 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.931124 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:03.941865 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:03.949908 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.950619 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:03.971668 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:03.979712 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:03.980537 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:03.983949 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.009232 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:04.009533 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:04.045182 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:04.109002 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.118583 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:04.118583 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:04.124373 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:04.126638 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.131618 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.132906 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:04.134654 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:04.146605 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:04.151176 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:04.166882 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:04.174612 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:04.177667 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.181311 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:04.249799 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:04.271160 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:04.276420 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:04.279160 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:04.281430 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.285151 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.292613 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:04.296927 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:04.365399 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:04.477711 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:04.479436 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:04.480661 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.486725 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:04.489218 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:04.493099 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:04.503281 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:04.623721 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.624358 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:04.635841 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:04.639434 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:04.639513 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:04.647891 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.648192 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:04.663221 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:04.667824 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.680027 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:04.680045 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:04.682763 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:04.685392 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.691855 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:04.703809 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:02:04.762513 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:02:04.780644 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:02:04.807479 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:04.808144 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:04.809764 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:04.810187 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.811192 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.829339 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:04.832583 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:04.886673 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:02:04.897480 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:02:04.902693 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:04.917140 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:04.921020 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:04.922506 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.922866 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:04.923395 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:04.925575 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:04.989074 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:05.030658 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:05.034072 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:05.037016 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:05.037891 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:05.048126 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:05.061728 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.069236 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:05.088392 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:05.089143 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:05.089527 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:05.092075 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.102885 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:05.103446 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:05.143579 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:05.158494 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:05.205492 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:05.215854 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:05.217145 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.226521 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.230156 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:05.230156 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:05.243538 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:05.244670 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:05.257769 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.260442 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:05.263630 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:05.279835 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:05.288031 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:02:05.288172 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:05.290467 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:05.312387 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:02:05.350847 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240503 14:02:05.352459 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:02:05.358822 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:02:05.398801 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:02:05.519659 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:05.557201 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:05.593853 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:05.597242 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:05.605444 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:05.613152 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:05.615715 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:05.623389 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:05.630039 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:05.634377 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.650957 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.651489 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.658252 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.659097 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:05.710096 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:02:05.728883 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:05.776367 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:05.784421 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:05.786825 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.787155 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:05.789908 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.798097 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:05.798298 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.812114 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:05.812538 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:05.814038 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:05.820261 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.829165 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:05.829416 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:05.829649 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:05.872388 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 14:02:05.873896 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:02:05.876266 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:05.936976 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:05.937309 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:05.937628 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.943429 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:05.945056 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:05.952040 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:05.983969 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:06.056247 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:06.057287 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:06.063100 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:06.064536 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.065845 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:06.066606 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:06.095801 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:06.168584 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:06.173621 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:06.176451 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.191349 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:06.198313 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:06.202417 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.239053 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.239347 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:06.239989 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:06.242266 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:06.246443 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.256070 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:06.269479 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:06.338155 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.339310 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:06.341220 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:06.342533 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:06.343668 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.359421 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:06.389281 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:06.470014 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:06.557521 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:06.557812 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.557865 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:06.560663 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.567896 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:06.571302 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:06.586977 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:02:06.588780 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:06.686017 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:06.688527 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:06.690421 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:06.694483 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:06.698482 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:06.708081 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:06.708922 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.709806 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.711105 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.712756 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:06.717082 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:06.717608 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:06.720208 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:06.742460 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.749303 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:06.764575 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 14:02:06.792129 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:06.873467 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:06.873533 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.874892 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:06.888135 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:06.888722 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:06.891158 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.905694 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:06.908201 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.915218 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:06.916194 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:06.931329 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:06.933811 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:06.944263 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:06.949599 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:07.043084 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.043507 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:07.046943 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:07.049039 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.056852 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:07.062532 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:07.149140 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:07.149892 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:07.150871 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:07.159893 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:07.161221 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:07.163820 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.167722 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.175436 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:07.202150 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:02:07.202309 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:02:07.216382 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:02:07.253526 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:02:07.269369 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:07.270120 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:02:07.272491 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:07.272521 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.273121 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:07.274653 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:07.296984 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:07.375376 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:07.391935 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:07.398721 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:07.398766 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:07.400143 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:07.400790 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.401139 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:07.403396 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:07.446826 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:07.491762 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:07.492273 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:07.492602 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:07.496560 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:07.497474 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.503391 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:07.503414 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:07.589645 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:02:07.592207 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:07.607627 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:07.609614 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:07.611841 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:07.623257 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:07.625293 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.626399 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:07.682500 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:07.715653 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:07.717417 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.720912 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:07.720932 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:07.738585 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:07.739033 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:07.745700 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:07.775404 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:07.790534 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.790674 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:07.799969 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:07.802693 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:07.809273 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:07.819943 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.850492 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(f5c06806a19e4f4592d26ed4acc651cd): perf score=0.033371
W20240503 14:02:07.853657 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:07.890434 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(f5c06806a19e4f4592d26ed4acc651cd) complete. Timing: real 0.039s user 0.034s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":2521,"dirs.queue_time_us":10801,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1375,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":604,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":226,"threads_started":3,"wal-append.queue_time_us":336}
I20240503 14:02:07.891701 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(f5c06806a19e4f4592d26ed4acc651cd): 4838 bytes on disk
W20240503 14:02:07.892127 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:07.892941 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(f5c06806a19e4f4592d26ed4acc651cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240503 14:02:07.904327 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:07.905153 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:07.913368 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:07.914551 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(6c4f3bc64e7145e2b692beab730d1496): perf score=0.033389
I20240503 14:02:07.919121 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:07.927383 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:07.934617 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:07.952556 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:07.968801 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:07.982185 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:08.006888 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(6c4f3bc64e7145e2b692beab730d1496) complete. Timing: real 0.092s user 0.024s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":17748,"dirs.queue_time_us":16574,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":6779,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":614,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":275,"threads_started":3,"wal-append.queue_time_us":11223}
I20240503 14:02:08.008495 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(6c4f3bc64e7145e2b692beab730d1496): 4921 bytes on disk
I20240503 14:02:08.009449 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(6c4f3bc64e7145e2b692beab730d1496) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 14:02:08.015084 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:08.018500 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:08.018702 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.020959 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:08.024457 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.041873 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:08.065435 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:02:08.071897 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:08.135066 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:02:08.164232 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:08.176455 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:08.177474 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:08.178180 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:08.181023 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.188794 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.193516 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:08.246759 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(f4f9cb87c70e43dbb0d850c275ebb8a5): perf score=0.033363
I20240503 14:02:08.265058 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(f4f9cb87c70e43dbb0d850c275ebb8a5): perf score=0.033365
I20240503 14:02:08.292413 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(f4f9cb87c70e43dbb0d850c275ebb8a5) complete. Timing: real 0.045s user 0.032s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":2927,"dirs.queue_time_us":182,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1108,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":652,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":296,"threads_started":3,"wal-append.queue_time_us":181}
I20240503 14:02:08.293589 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(f4f9cb87c70e43dbb0d850c275ebb8a5): 4836 bytes on disk
I20240503 14:02:08.294198 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(f4f9cb87c70e43dbb0d850c275ebb8a5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 14:02:08.296614 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(f4f9cb87c70e43dbb0d850c275ebb8a5) complete. Timing: real 0.031s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1150,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":549,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":106,"threads_started":1,"wal-append.queue_time_us":181}
I20240503 14:02:08.297433 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(f4f9cb87c70e43dbb0d850c275ebb8a5): 4836 bytes on disk
I20240503 14:02:08.307706 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(f4f9cb87c70e43dbb0d850c275ebb8a5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240503 14:02:08.328521 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:08.355187 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:08.358932 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:08.358950 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.362555 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.369696 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:08.376121 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(a0106f7a40cb467eb4738890ac4e22d4): perf score=0.033398
W20240503 14:02:08.384460 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:02:08.386736 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:08.435938 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(a0106f7a40cb467eb4738890ac4e22d4) complete. Timing: real 0.056s user 0.034s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":47176,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":608,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":265,"threads_started":3,"wal-append.queue_time_us":184}
I20240503 14:02:08.437268 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(a0106f7a40cb467eb4738890ac4e22d4): 4923 bytes on disk
I20240503 14:02:08.437844 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(a0106f7a40cb467eb4738890ac4e22d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240503 14:02:08.438642 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(c2e9f6eecbec45988b290775277dbddc): perf score=0.033412
W20240503 14:02:08.453562 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:08.469975 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(c2e9f6eecbec45988b290775277dbddc) complete. Timing: real 0.031s user 0.020s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1453,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":646,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":88,"threads_started":1,"wal-append.queue_time_us":9012}
I20240503 14:02:08.470907 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(c2e9f6eecbec45988b290775277dbddc): 4583 bytes on disk
I20240503 14:02:08.471360 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(c2e9f6eecbec45988b290775277dbddc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240503 14:02:08.508185 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:02:08.515111 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:08.579483 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:08.579519 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:08.579558 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.585264 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.588984 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:08.596561 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:08.611063 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:08.612848 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:08.615541 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.616425 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:08.637554 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:08.645351 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:08.652419 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.781860 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:08.782122 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.787393 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.788717 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:08.794948 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:08.796483 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(7d2e1517a8ac4041bfb4b90634cb0fa6): perf score=0.033335
I20240503 14:02:08.809126 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:08.810847 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(7d2e1517a8ac4041bfb4b90634cb0fa6): perf score=0.033338
I20240503 14:02:08.831147 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(7d2e1517a8ac4041bfb4b90634cb0fa6) complete. Timing: real 0.034s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":2433,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":492,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":118,"threads_started":2,"wal-append.queue_time_us":130}
I20240503 14:02:08.831986 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(7d2e1517a8ac4041bfb4b90634cb0fa6): 4923 bytes on disk
I20240503 14:02:08.833539 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(7d2e1517a8ac4041bfb4b90634cb0fa6) complete. Timing: real 0.021s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":180,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":2410,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":419,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":6714}
I20240503 14:02:08.834103 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(7d2e1517a8ac4041bfb4b90634cb0fa6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 14:02:08.834342 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(11a8c270532947ba9b505aace44b56f6): perf score=0.033343
I20240503 14:02:08.834787 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(d1e9155230c74e92b6ee441de713a419): perf score=0.033345
W20240503 14:02:08.837145 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:08.851490 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(d1e9155230c74e92b6ee441de713a419) complete. Timing: real 0.016s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":989,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":477,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":201}
I20240503 14:02:08.851538 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(11a8c270532947ba9b505aace44b56f6) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1376,"drs_written":1,"lbm_read_time_us":32,"lbm_reads_lt_1ms":4,"lbm_write_time_us":425,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":139}
I20240503 14:02:08.852348 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(d1e9155230c74e92b6ee441de713a419): 4583 bytes on disk
I20240503 14:02:08.852361 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(7d2e1517a8ac4041bfb4b90634cb0fa6): 4923 bytes on disk
I20240503 14:02:08.852906 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(d1e9155230c74e92b6ee441de713a419) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 14:02:08.853319 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(7d2e1517a8ac4041bfb4b90634cb0fa6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 14:02:08.853997 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(11a8c270532947ba9b505aace44b56f6): 4837 bytes on disk
I20240503 14:02:08.854317 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(11a8c270532947ba9b505aace44b56f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240503 14:02:08.896970 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(11a8c270532947ba9b505aace44b56f6): perf score=0.033363
I20240503 14:02:08.917821 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:08.922179 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:08.923761 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:08.931471 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:08.935710 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:08.936587 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:08.947129 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(11a8c270532947ba9b505aace44b56f6) complete. Timing: real 0.050s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4051,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":2382,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":689,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":28096}
I20240503 14:02:08.948168 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(11a8c270532947ba9b505aace44b56f6): 4837 bytes on disk
I20240503 14:02:08.948621 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(11a8c270532947ba9b505aace44b56f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 14:02:08.960450 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:08.976815 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:08.977007 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(d1e9155230c74e92b6ee441de713a419): perf score=0.033383
I20240503 14:02:08.991461 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(d1e9155230c74e92b6ee441de713a419) complete. Timing: real 0.014s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1489,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":488,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":120}
I20240503 14:02:08.992441 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(d1e9155230c74e92b6ee441de713a419): 4583 bytes on disk
I20240503 14:02:08.992811 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(d1e9155230c74e92b6ee441de713a419) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 14:02:09.033773 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:09.035153 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.038810 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:09.041013 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:09.053848 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:09.055400 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:09.078209 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.114284 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:09.115679 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.116155 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:09.117718 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:09.118090 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:09.119777 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.211387 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:09.215610 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:09.223042 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:09.224500 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:09.234477 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.240559 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:09.242074 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:09.275384 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:09.291600 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:09.294145 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:09.296097 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.296473 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.300688 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:09.312649 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:09.313030 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:09.349166 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:02:09.355556 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:09.450758 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(4a608f683c4f41f3bbdfc9b4c219ce49): perf score=0.033343
I20240503 14:02:09.462728 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(4a608f683c4f41f3bbdfc9b4c219ce49) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1661,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":474,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":174,"threads_started":2,"wal-append.queue_time_us":157}
I20240503 14:02:09.463629 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(4a608f683c4f41f3bbdfc9b4c219ce49): 4838 bytes on disk
I20240503 14:02:09.464047 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(4a608f683c4f41f3bbdfc9b4c219ce49) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 14:02:09.494874 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(adf2c44328e244779955e712b99f8331): perf score=0.033352
I20240503 14:02:09.509590 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(adf2c44328e244779955e712b99f8331) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":137,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1716,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":130}
I20240503 14:02:09.510406 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(adf2c44328e244779955e712b99f8331): 4923 bytes on disk
I20240503 14:02:09.510735 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(adf2c44328e244779955e712b99f8331) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240503 14:02:09.530994 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:02:09.535463 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:09.536367 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:09.541409 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:09.543289 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.544898 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.547432 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:09.551669 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:09.606086 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(4a608f683c4f41f3bbdfc9b4c219ce49): perf score=0.033384
I20240503 14:02:09.607558 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(adf2c44328e244779955e712b99f8331): perf score=0.033387
I20240503 14:02:09.625860 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(4a608f683c4f41f3bbdfc9b4c219ce49) complete. Timing: real 0.020s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3860,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":621,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":155,"threads_started":2,"wal-append.queue_time_us":222}
I20240503 14:02:09.626754 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(4a608f683c4f41f3bbdfc9b4c219ce49): 4838 bytes on disk
I20240503 14:02:09.627199 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(4a608f683c4f41f3bbdfc9b4c219ce49) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240503 14:02:09.643074 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(adf2c44328e244779955e712b99f8331) complete. Timing: real 0.035s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1694,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1427,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":174}
I20240503 14:02:09.644796 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(211de6f35a844e278e8c1eb1621e7c5f): perf score=0.033395
I20240503 14:02:09.651157 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:09.652400 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:09.655560 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:09.671947 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:09.675894 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.676612 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:09.678115 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:09.681866 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(211de6f35a844e278e8c1eb1621e7c5f) complete. Timing: real 0.037s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1577,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":627,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":11519}
I20240503 14:02:09.682559 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(adf2c44328e244779955e712b99f8331): 4923 bytes on disk
I20240503 14:02:09.682898 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(adf2c44328e244779955e712b99f8331) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240503 14:02:09.683562 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(211de6f35a844e278e8c1eb1621e7c5f): 4583 bytes on disk
I20240503 14:02:09.683924 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(211de6f35a844e278e8c1eb1621e7c5f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240503 14:02:09.729645 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:09.730361 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:09.733500 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:09.735417 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.742446 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:09.759826 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:09.807996 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:09.826834 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:09.830570 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:09.832321 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:09.833352 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.836043 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:09.846704 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:09.862608 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:09.872699 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:02:09.878890 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:09.878926 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(faa0a6ca00d442eaaf9fca7fc1259ec7): perf score=0.033345
I20240503 14:02:09.892397 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(faa0a6ca00d442eaaf9fca7fc1259ec7) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1627,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":469,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":145}
I20240503 14:02:09.893086 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(faa0a6ca00d442eaaf9fca7fc1259ec7): 4583 bytes on disk
I20240503 14:02:09.893479 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(faa0a6ca00d442eaaf9fca7fc1259ec7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240503 14:02:09.898731 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:02:09.902974 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:02:09.917333 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:09.935772 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(c42fee9fadbb4601a7fe32eb6a04e767): perf score=0.033363
W20240503 14:02:09.937419 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:09.944289 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:09.946038 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.946478 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:09.948839 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:09.959713 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:09.961779 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(c42fee9fadbb4601a7fe32eb6a04e767) complete. Timing: real 0.025s user 0.015s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":282,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":4918,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":595,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":154}
I20240503 14:02:09.962639 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(c42fee9fadbb4601a7fe32eb6a04e767): 4923 bytes on disk
I20240503 14:02:09.963033 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(c42fee9fadbb4601a7fe32eb6a04e767) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 14:02:09.982039 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:10.013171 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(a6a89230883347d4a64d96d321247012): perf score=0.033381
I20240503 14:02:10.030429 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(a6a89230883347d4a64d96d321247012) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":156,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":2713,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":524,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":121}
I20240503 14:02:10.031352 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(a6a89230883347d4a64d96d321247012): 4838 bytes on disk
I20240503 14:02:10.031800 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(a6a89230883347d4a64d96d321247012) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 14:02:10.069546 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.069653 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:10.071928 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:10.082701 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:10.084169 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.094429 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:10.132858 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:10.137687 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.145628 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:10.147413 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:10.148030 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:10.149724 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:10.197664 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:10.198143 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:10.204038 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:02:10.243973 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:10.267140 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:10.268445 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.277258 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.277369 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:10.281474 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:10.298274 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.299203 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:10.305105 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:10.306435 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.320721 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:10.327003 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:10.330374 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:10.341737 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:02:10.356865 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240503 14:02:10.395843 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:02:10.407716 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:10.479465 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:10.638422 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:10.642827 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.646407 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(d837845c6f644bca83315c9f7c45246d): perf score=0.033356
I20240503 14:02:10.647817 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:10.650558 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:10.659442 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:10.660415 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:10.666929 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:10.676714 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(d837845c6f644bca83315c9f7c45246d) complete. Timing: real 0.030s user 0.013s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5523,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":577,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":275}
I20240503 14:02:10.677632 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(d837845c6f644bca83315c9f7c45246d): 4838 bytes on disk
I20240503 14:02:10.678103 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(d837845c6f644bca83315c9f7c45246d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240503 14:02:10.680234 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.690944 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:10.703181 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:10.705237 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.708163 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:10.715757 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(d837845c6f644bca83315c9f7c45246d): perf score=0.033374
W20240503 14:02:10.716601 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:10.720044 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(355b5456887b4c5694badc9d53b1809a): perf score=0.033376
I20240503 14:02:10.723560 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.738513 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(d837845c6f644bca83315c9f7c45246d) complete. Timing: real 0.023s user 0.013s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7437,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":556,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":3670}
I20240503 14:02:10.739532 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(d837845c6f644bca83315c9f7c45246d): 4838 bytes on disk
I20240503 14:02:10.740446 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(d837845c6f644bca83315c9f7c45246d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240503 14:02:10.749697 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:10.756639 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(355b5456887b4c5694badc9d53b1809a) complete. Timing: real 0.036s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":2581,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":509,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":183}
I20240503 14:02:10.757733 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(355b5456887b4c5694badc9d53b1809a): 4583 bytes on disk
I20240503 14:02:10.758324 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(355b5456887b4c5694badc9d53b1809a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 14:02:10.759652 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:10.777868 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:10.785477 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(79b21b1e5292488aad442060b417f3c1): perf score=0.033393
W20240503 14:02:10.785845 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:02:10.795207 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:10.803673 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(79b21b1e5292488aad442060b417f3c1) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":571,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":119}
I20240503 14:02:10.804508 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(79b21b1e5292488aad442060b417f3c1): 4923 bytes on disk
I20240503 14:02:10.804967 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(79b21b1e5292488aad442060b417f3c1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240503 14:02:10.828341 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:10.845929 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:10.849709 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:10.853173 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.853863 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:10.863430 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.877480 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:10.891664 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:02:10.892307 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:10.914186 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:10.921734 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:10.922353 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.926909 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:10.929255 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:10.932010 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:10.947369 19308 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:02:10.947494 19308 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:02:10.947912 19308 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:02:10.948899 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:02:10.949033 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:02:10.949092 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:02:10.949851 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:10.949932 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:10.949999 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:02:10.950349 19308 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:02:11.013818 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:11.015831 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.017590 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:11.023391 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.024480 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:11.046190 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:11.056880 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(c8933689df2c45b59a9cd16024ddb7d9): perf score=0.033347
I20240503 14:02:11.071069 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(c8933689df2c45b59a9cd16024ddb7d9) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1556,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":497,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":694}
I20240503 14:02:11.071934 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(c8933689df2c45b59a9cd16024ddb7d9): 4582 bytes on disk
I20240503 14:02:11.072321 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(c8933689df2c45b59a9cd16024ddb7d9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240503 14:02:11.130206 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:11.130529 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:11.131872 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:11.136803 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:11.143059 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.143570 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:11.160676 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.180249 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(a4b4e16598b740cb9d690c2442a3e994): perf score=0.033382
I20240503 14:02:11.200096 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(a4b4e16598b740cb9d690c2442a3e994) complete. Timing: real 0.019s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1649,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":557,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":123}
I20240503 14:02:11.201206 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(a4b4e16598b740cb9d690c2442a3e994): 4923 bytes on disk
I20240503 14:02:11.201778 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(a4b4e16598b740cb9d690c2442a3e994) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 14:02:11.202461 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(4224ca4d93ce4784b52a7550468ce1b1): perf score=0.033386
I20240503 14:02:11.218456 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(4224ca4d93ce4784b52a7550468ce1b1) complete. Timing: real 0.016s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1861,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":141}
I20240503 14:02:11.219367 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(4224ca4d93ce4784b52a7550468ce1b1): 4837 bytes on disk
I20240503 14:02:11.219755 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(4224ca4d93ce4784b52a7550468ce1b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 14:02:11.260731 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(4224ca4d93ce4784b52a7550468ce1b1): perf score=0.033396
W20240503 14:02:11.260988 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:11.279181 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:11.285737 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:11.292292 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.292337 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.298050 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:11.303886 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:11.305198 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(4224ca4d93ce4784b52a7550468ce1b1) complete. Timing: real 0.044s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":9447,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":2920,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1930,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":144}
I20240503 14:02:11.307957 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(4224ca4d93ce4784b52a7550468ce1b1): 4837 bytes on disk
I20240503 14:02:11.309118 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:11.309194 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(4224ca4d93ce4784b52a7550468ce1b1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 14:02:11.310597 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(c8933689df2c45b59a9cd16024ddb7d9): perf score=0.033409
W20240503 14:02:11.326817 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:11.327225 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:11.331545 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:11.334286 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(c8933689df2c45b59a9cd16024ddb7d9) complete. Timing: real 0.023s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":2029,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":499,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":105,"threads_started":1,"wal-append.queue_time_us":6731}
I20240503 14:02:11.335104 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(c8933689df2c45b59a9cd16024ddb7d9): 4582 bytes on disk
I20240503 14:02:11.335549 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(c8933689df2c45b59a9cd16024ddb7d9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 14:02:11.340274 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.340356 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.340497 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:11.346575 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:11.414633 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:11.461725 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:11.462178 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.466300 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:11.467447 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:11.474403 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:11.474452 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:11.564975 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:02:11.600425 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:02:11.665577 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:11.669340 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.669869 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:11.671749 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.673594 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:11.674892 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:11.683503 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:11.718643 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:02:11.741292 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:11.747437 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(2089eeda1e0f4ff5926584d472e6631d): perf score=0.033361
I20240503 14:02:11.765707 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(2089eeda1e0f4ff5926584d472e6631d) complete. Timing: real 0.018s user 0.012s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3498,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1732,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":260}
I20240503 14:02:11.766475 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(df13ec098dc548afac8c67195d25e7e8): perf score=0.033364
I20240503 14:02:11.786435 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(df13ec098dc548afac8c67195d25e7e8) complete. Timing: real 0.020s user 0.011s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1907,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":588,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":2324}
I20240503 14:02:11.787542 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(2089eeda1e0f4ff5926584d472e6631d): 4923 bytes on disk
I20240503 14:02:11.787914 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(2089eeda1e0f4ff5926584d472e6631d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 14:02:11.788595 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(df13ec098dc548afac8c67195d25e7e8): 4586 bytes on disk
I20240503 14:02:11.788918 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(df13ec098dc548afac8c67195d25e7e8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 14:02:11.795787 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:11.798370 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:11.806684 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.806968 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:11.817014 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:11.817667 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.827597 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(df13ec098dc548afac8c67195d25e7e8): perf score=0.033378
I20240503 14:02:11.837682 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(70ffc5888a5945d6a8e166420e6f1bd9): perf score=0.033382
W20240503 14:02:11.849123 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:02:11.855566 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(df13ec098dc548afac8c67195d25e7e8) complete. Timing: real 0.028s user 0.014s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":16622,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":549,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":169}
I20240503 14:02:11.856488 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(df13ec098dc548afac8c67195d25e7e8): 4586 bytes on disk
I20240503 14:02:11.856904 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(df13ec098dc548afac8c67195d25e7e8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 14:02:11.862963 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:11.868221 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:11.869318 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:11.872601 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(70ffc5888a5945d6a8e166420e6f1bd9) complete. Timing: real 0.033s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3637,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":539,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":123,"threads_started":2,"wal-append.queue_time_us":251}
I20240503 14:02:11.873495 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(70ffc5888a5945d6a8e166420e6f1bd9): 4838 bytes on disk
I20240503 14:02:11.873900 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(70ffc5888a5945d6a8e166420e6f1bd9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240503 14:02:11.875576 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:11.891664 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.893249 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:11.908130 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:11.910282 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:11.917331 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:12.002919 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:12.011421 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.018257 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:12.021454 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:12.027874 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:12.039824 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:12.047358 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:12.059134 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.067315 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:12.070521 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.071326 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:12.082110 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:12.095533 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:12.169263 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:12.173575 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:12.176435 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.178447 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:12.192139 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:12.198936 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:12.202939 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:12.223093 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(dcdc3ae5232a49cdb003d9a2e3cf6d92): perf score=0.033361
I20240503 14:02:12.238762 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(dcdc3ae5232a49cdb003d9a2e3cf6d92) complete. Timing: real 0.015s user 0.011s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1572,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":512,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":133}
I20240503 14:02:12.239738 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(dcdc3ae5232a49cdb003d9a2e3cf6d92): 4923 bytes on disk
I20240503 14:02:12.240064 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(dcdc3ae5232a49cdb003d9a2e3cf6d92) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240503 14:02:12.254663 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240503 14:02:12.271742 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:12.291006 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(b031f09b97664d1a94c14f0661181899): perf score=0.033375
I20240503 14:02:12.293762 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:12.296845 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:12.298226 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.303439 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.313020 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(b031f09b97664d1a94c14f0661181899) complete. Timing: real 0.022s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1513,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":982}
I20240503 14:02:12.313843 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(b031f09b97664d1a94c14f0661181899): 4584 bytes on disk
I20240503 14:02:12.314262 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(b031f09b97664d1a94c14f0661181899) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 14:02:12.315822 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:12.320340 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:12.322218 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:12.359095 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(b031f09b97664d1a94c14f0661181899): perf score=0.033396
I20240503 14:02:12.376411 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(dcdc3ae5232a49cdb003d9a2e3cf6d92): perf score=0.033403
I20240503 14:02:12.377085 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(b031f09b97664d1a94c14f0661181899) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":148,"dirs.run_cpu_time_us":276,"dirs.run_wall_time_us":1495,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":491,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":144}
I20240503 14:02:12.377995 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(b031f09b97664d1a94c14f0661181899): 4584 bytes on disk
I20240503 14:02:12.378350 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(b031f09b97664d1a94c14f0661181899) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 14:02:12.393080 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(dcdc3ae5232a49cdb003d9a2e3cf6d92) complete. Timing: real 0.016s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":254,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1849,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":657,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":195,"threads_started":2,"wal-append.queue_time_us":199}
I20240503 14:02:12.394134 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(3481715d755a4277b0905b0832fad4af): perf score=0.033405
W20240503 14:02:12.397300 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:12.403707 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:02:12.408528 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:12.421418 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(3481715d755a4277b0905b0832fad4af) complete. Timing: real 0.027s user 0.012s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1528,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":554,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":141}
I20240503 14:02:12.422767 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(dcdc3ae5232a49cdb003d9a2e3cf6d92): 4923 bytes on disk
I20240503 14:02:12.423305 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(dcdc3ae5232a49cdb003d9a2e3cf6d92) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240503 14:02:12.424253 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(3481715d755a4277b0905b0832fad4af): 4838 bytes on disk
I20240503 14:02:12.424487 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:12.424650 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:12.424626 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(3481715d755a4277b0905b0832fad4af) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 14:02:12.425179 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:12.425822 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:12.432421 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:12.435360 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.435599 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:12.462584 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:12.535552 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:12.539490 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:12.541746 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:12.542142 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.542574 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:12.545006 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:02:12.552538 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:12.568222 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling FlushMRSOp(0e442bb664f44887aa8afdb62e1d97fc): perf score=0.033350
I20240503 14:02:12.597930 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: FlushMRSOp(0e442bb664f44887aa8afdb62e1d97fc) complete. Timing: real 0.029s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1410,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":733,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":104,"threads_started":1,"wal-append.queue_time_us":196}
I20240503 14:02:12.598719 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(0e442bb664f44887aa8afdb62e1d97fc): 4838 bytes on disk
I20240503 14:02:12.599103 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(0e442bb664f44887aa8afdb62e1d97fc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 14:02:12.629719 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(e92d8f55591346158b9583feb958a8ca): perf score=0.033368
I20240503 14:02:12.652410 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:12.652962 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.652956 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:12.653666 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(e92d8f55591346158b9583feb958a8ca) complete. Timing: real 0.024s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":403,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1973,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":70,"threads_started":1,"wal-append.queue_time_us":2938}
I20240503 14:02:12.654552 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(e92d8f55591346158b9583feb958a8ca): 4922 bytes on disk
I20240503 14:02:12.655478 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(e92d8f55591346158b9583feb958a8ca) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 14:02:12.669176 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:12.670001 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.670641 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:12.742080 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(0e442bb664f44887aa8afdb62e1d97fc): perf score=0.033396
I20240503 14:02:12.761565 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(0e442bb664f44887aa8afdb62e1d97fc) complete. Timing: real 0.019s user 0.014s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3363,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1399,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":173,"threads_started":2,"wal-append.queue_time_us":294}
I20240503 14:02:12.762295 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(0e442bb664f44887aa8afdb62e1d97fc): 4838 bytes on disk
I20240503 14:02:12.762687 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(0e442bb664f44887aa8afdb62e1d97fc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 14:02:12.777865 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.779143 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:12.781531 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:12.785142 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:12.789183 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.789471 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:12.809448 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:02:12.865172 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:12.890462 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:02:12.894832 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:12.895889 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:12.896847 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:12.905247 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:12.906260 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:12.908972 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.917246 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:12.919384 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:12.959704 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:12.960057 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:12.961601 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:12.964120 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:12.966636 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.981058 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:12.981235 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:13.024715 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:13.083961 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:13.093587 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:13.094461 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:13.098321 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.114604 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.116055 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:13.121801 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:13.137151 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:13.138403 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.141130 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:13.147400 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:13.154654 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:13.203281 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:02:13.234923 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:13.254889 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:13.257822 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:13.261915 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:13.263104 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:13.265655 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:13.266314 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.290078 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:13.336612 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:02:13.367815 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:13.385094 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:13.453860 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.454111 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:13.454655 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:13.454949 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:13.455031 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:13.455626 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.460186 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:13.463644 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:02:13.466172 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:13.567552 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:13.568670 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:13.570480 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:13.576965 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.577512 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.591183 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:13.612048 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:13.625866 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:13.629092 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:13.633929 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:13.639890 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:13.643548 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.659386 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:13.690258 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:13.747679 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:13.751762 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:13.751904 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:13.752223 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.753250 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.758769 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:13.843490 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:13.854945 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:13.864059 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:13.864450 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:13.869526 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:13.871706 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:13.876770 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:13.895946 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:13.920298 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:02:13.967903 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:13.978345 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:13.989502 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:13.994441 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:13.995402 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:14.005329 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:14.007340 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.022748 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:14.023121 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:14.042784 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:14.051414 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:14.052703 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.054499 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.057823 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:14.062367 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:14.160619 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.160761 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:14.161504 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:14.162427 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:14.175021 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:14.192386 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.222375 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:14.226560 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:14.229306 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.237133 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:14.244217 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:14.251833 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.255594 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:14.274789 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:14.285550 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:02:14.396351 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:14.400179 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:02:14.401223 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:02:14.438922 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:02:14.505718 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:14.509480 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:14.511874 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.512092 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:14.527048 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.537557 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:14.541424 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:14.591558 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:14.600282 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:14.602125 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:14.609072 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.610453 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.634135 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:14.704020 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:14.707679 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.708813 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.715924 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:14.732690 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:14.752928 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:14.793699 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:14.796226 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:14.798182 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:14.799381 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:14.808588 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:02:14.814011 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:14.825747 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.828732 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:14.835593 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:14.842062 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:14.868592 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:14.882102 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:14.913620 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:14.913790 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:14.917162 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.917469 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:14.928121 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:14.941431 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:15.037348 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:15.043143 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.046178 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:15.056413 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:15.058116 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:15.063699 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:15.090162 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:15.105162 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.110764 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:15.111289 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.113191 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:15.127948 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:15.137856 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:15.148857 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:02:15.221508 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.225654 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.226119 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:15.229099 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:15.246475 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:15.246773 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:15.260725 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:15.265965 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:15.284336 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:15.295370 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.295367 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:15.305357 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:15.315994 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:02:15.316699 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:15.319473 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:15.360245 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240503 14:02:15.360970 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:02:15.366686 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:15.401197 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:02:15.532388 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:15.593402 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:15.596724 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:15.598287 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.603399 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.613579 31688 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:02:15.613691 31688 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:02:15.614063 31688 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:02:15.615492 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:15.616685 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:15.616765 12103 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:02:15.616905 31688 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:02:15.618623 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:15.618748 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:15.637712 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:15.639894 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:15.648394 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.648651 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:15.649672 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:15.659061 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:15.673867 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:15.684312 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:15.699942 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:02:15.728637 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:15.770650 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:15.775128 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:15.780042 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.787534 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:15.792652 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:15.796574 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:15.801355 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:15.813302 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.830121 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:15.836356 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.843592 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:15.847201 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:15.851673 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.852123 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:15.857657 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:15.863364 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:02:15.886463 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:15.942941 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.944181 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:15.944612 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:15.944779 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:15.946411 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:15.979739 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:15.984313 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:02:15.989988 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:02:16.009747 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 14:02:16.062487 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:16.062969 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.063447 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:16.068194 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:16.086378 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:16.086772 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.174739 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:16.179735 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:16.185873 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.187638 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.201297 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:16.202883 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:16.235385 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:16.246336 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.247090 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:16.249823 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:16.256261 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:16.266960 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:16.267133 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:16.284619 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:16.345780 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:16.346017 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:16.346262 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:16.355906 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.356755 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.357141 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:16.449796 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:16.552961 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:16.559437 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.559970 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:16.563513 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:16.563599 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:16.584698 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:16.586704 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:16.594365 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:02:16.613493 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:16.681907 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:16.688344 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.690423 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:16.697733 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.708035 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:16.709364 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(2089eeda1e0f4ff5926584d472e6631d): perf score=0.033379
I20240503 14:02:16.715308 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:16.718551 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.724483 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:16.745002 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.751627 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:16.764676 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:16.776388 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(92506cde7b084b14b88d4a915731eeb3): perf score=0.033376
I20240503 14:02:16.779783 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:16.808316 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:16.822479 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(2089eeda1e0f4ff5926584d472e6631d) complete. Timing: real 0.113s user 0.032s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":60639,"dirs.queue_time_us":30225,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1225,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":514,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":202,"threads_started":3,"wal-append.queue_time_us":2792}
I20240503 14:02:16.824249 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(dcdc3ae5232a49cdb003d9a2e3cf6d92): perf score=0.033407
W20240503 14:02:16.843145 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:16.849112 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(dcdc3ae5232a49cdb003d9a2e3cf6d92) complete. Timing: real 0.025s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":190,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1810,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":557,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":87,"threads_started":1,"wal-append.queue_time_us":1789}
I20240503 14:02:16.850023 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(2089eeda1e0f4ff5926584d472e6631d): 4923 bytes on disk
I20240503 14:02:16.849714 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(92506cde7b084b14b88d4a915731eeb3) complete. Timing: real 0.073s user 0.023s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":23886,"dirs.queue_time_us":2006,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1072,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4957,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":244,"threads_started":3,"wal-append.queue_time_us":4233}
I20240503 14:02:16.850586 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(2089eeda1e0f4ff5926584d472e6631d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 14:02:16.851176 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(92506cde7b084b14b88d4a915731eeb3): 4583 bytes on disk
I20240503 14:02:16.851358 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(dcdc3ae5232a49cdb003d9a2e3cf6d92): 4923 bytes on disk
I20240503 14:02:16.851660 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(dcdc3ae5232a49cdb003d9a2e3cf6d92) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 14:02:16.851703 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(92506cde7b084b14b88d4a915731eeb3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240503 14:02:16.852303 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(6c4f3bc64e7145e2b692beab730d1496): perf score=0.033354
I20240503 14:02:16.852414 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(79b21b1e5292488aad442060b417f3c1): perf score=0.033362
W20240503 14:02:16.853646 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:16.884279 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(6c4f3bc64e7145e2b692beab730d1496) complete. Timing: real 0.032s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1472,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":520,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":126,"threads_started":1,"wal-append.queue_time_us":11586}
I20240503 14:02:16.885257 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(7d2e1517a8ac4041bfb4b90634cb0fa6): perf score=0.033358
W20240503 14:02:16.885581 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:02:16.897548 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:16.906258 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(79b21b1e5292488aad442060b417f3c1) complete. Timing: real 0.054s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":722,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1755,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":19558}
I20240503 14:02:16.907377 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(5e6ca408c4254678b67a74f49a34881b): perf score=0.033377
I20240503 14:02:16.908509 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(7d2e1517a8ac4041bfb4b90634cb0fa6) complete. Timing: real 0.023s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":2407,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":517,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":177}
W20240503 14:02:16.909478 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:16.909605 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(7d2e1517a8ac4041bfb4b90634cb0fa6): 4923 bytes on disk
I20240503 14:02:16.910002 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(7d2e1517a8ac4041bfb4b90634cb0fa6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240503 14:02:16.910766 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(6c4f3bc64e7145e2b692beab730d1496): 4921 bytes on disk
I20240503 14:02:16.912202 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(6c4f3bc64e7145e2b692beab730d1496) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240503 14:02:16.942122 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:16.970474 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(5e6ca408c4254678b67a74f49a34881b) complete. Timing: real 0.063s user 0.020s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1272,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":99,"threads_started":1,"wal-append.queue_time_us":24043}
I20240503 14:02:16.972347 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(79b21b1e5292488aad442060b417f3c1): 4923 bytes on disk
I20240503 14:02:16.972853 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(79b21b1e5292488aad442060b417f3c1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 14:02:16.973629 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(faa0a6ca00d442eaaf9fca7fc1259ec7): perf score=0.033379
I20240503 14:02:16.991715 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:16.991770 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:16.993901 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:16.994176 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:17.002664 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:17.008155 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:17.013432 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.032516 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:17.032523 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:17.036980 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.044852 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:17.046627 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.089947 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:17.133656 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:17.144197 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(faa0a6ca00d442eaaf9fca7fc1259ec7) complete. Timing: real 0.170s user 0.019s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1565,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":547,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":116,"threads_started":1,"wal-append.queue_time_us":794}
I20240503 14:02:17.145325 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(7d2e1517a8ac4041bfb4b90634cb0fa6): perf score=0.033394
I20240503 14:02:17.164067 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(7d2e1517a8ac4041bfb4b90634cb0fa6) complete. Timing: real 0.019s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1939,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":459,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":1792,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":230}
I20240503 14:02:17.164824 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(7d2e1517a8ac4041bfb4b90634cb0fa6): 4923 bytes on disk
I20240503 14:02:17.165194 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(7d2e1517a8ac4041bfb4b90634cb0fa6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 14:02:17.166553 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(c42fee9fadbb4601a7fe32eb6a04e767): perf score=0.033385
I20240503 14:02:17.175791 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(5e6ca408c4254678b67a74f49a34881b): 4583 bytes on disk
W20240503 14:02:17.177481 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:17.177953 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(5e6ca408c4254678b67a74f49a34881b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240503 14:02:17.178836 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(4a608f683c4f41f3bbdfc9b4c219ce49): perf score=0.033401
I20240503 14:02:17.196108 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(4a608f683c4f41f3bbdfc9b4c219ce49) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1283,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":416,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":162}
I20240503 14:02:17.197113 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(4a608f683c4f41f3bbdfc9b4c219ce49): 4838 bytes on disk
I20240503 14:02:17.197532 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(4a608f683c4f41f3bbdfc9b4c219ce49) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 14:02:17.198216 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(faa0a6ca00d442eaaf9fca7fc1259ec7): 4583 bytes on disk
I20240503 14:02:17.198518 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(faa0a6ca00d442eaaf9fca7fc1259ec7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4}
I20240503 14:02:17.200584 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(c42fee9fadbb4601a7fe32eb6a04e767) complete. Timing: real 0.033s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":1011,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":14426,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":99,"threads_started":1,"wal-append.queue_time_us":2109}
I20240503 14:02:17.201462 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(a4b4e16598b740cb9d690c2442a3e994): perf score=0.033364
I20240503 14:02:17.220348 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(a4b4e16598b740cb9d690c2442a3e994) complete. Timing: real 0.019s user 0.014s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":292,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1200,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":502,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":93,"threads_started":1,"wal-append.queue_time_us":557}
I20240503 14:02:17.221916 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(a4b4e16598b740cb9d690c2442a3e994): 4923 bytes on disk
I20240503 14:02:17.222496 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(a4b4e16598b740cb9d690c2442a3e994) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240503 14:02:17.223583 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(c42fee9fadbb4601a7fe32eb6a04e767): 4923 bytes on disk
I20240503 14:02:17.224489 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(c42fee9fadbb4601a7fe32eb6a04e767) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 14:02:17.228730 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:17.243335 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:17.243817 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.248477 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:17.249603 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.251356 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:17.256243 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:17.275863 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 14:02:17.304280 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:17.353497 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.355782 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:17.357419 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:17.358487 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.360116 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:17.375556 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:17.392850 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:17.397778 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:17.426680 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:17.450085 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(3481715d755a4277b0905b0832fad4af): perf score=0.033394
I20240503 14:02:17.453122 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:17.453864 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.456491 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:17.462353 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:17.464434 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:02:17.473330 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(3481715d755a4277b0905b0832fad4af) complete. Timing: real 0.022s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1511,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":619,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":148}
I20240503 14:02:17.474186 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(3481715d755a4277b0905b0832fad4af): 4838 bytes on disk
I20240503 14:02:17.474529 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(3481715d755a4277b0905b0832fad4af) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 14:02:17.476609 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:17.480458 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:17.564415 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:17.568917 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.569608 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:17.569780 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:17.578665 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:17.578941 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:17.604792 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:17.692116 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:17.694362 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:17.695870 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:17.698561 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.702838 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.711915 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:17.805191 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:17.808830 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.810806 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:17.811996 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:17.819386 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:17.820853 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.822229 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:17.831292 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:17.894243 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:17.909294 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:17.909811 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:17.912494 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:17.914041 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:17.919829 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:17.920202 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:17.936676 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:17.940431 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:17.974442 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:17.998683 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:17.998713 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:17.999157 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.006026 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.007814 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:18.014076 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:18.028909 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:02:18.095311 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:18.105535 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:18.105917 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:18.107806 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:18.108512 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.131912 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.145641 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:18.146689 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:18.161482 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:18.162770 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.172024 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:18.174083 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:18.177256 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:18.184085 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:18.206756 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:02:18.216996 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:18.227458 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(355b5456887b4c5694badc9d53b1809a): perf score=0.033377
I20240503 14:02:18.227922 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(0e442bb664f44887aa8afdb62e1d97fc): perf score=0.033388
I20240503 14:02:18.244738 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(355b5456887b4c5694badc9d53b1809a) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":212,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1564,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":579,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":187,"threads_started":2,"wal-append.queue_time_us":178}
I20240503 14:02:18.244830 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(0e442bb664f44887aa8afdb62e1d97fc) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":1270,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":545,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":309,"threads_started":2,"wal-append.queue_time_us":136}
I20240503 14:02:18.245594 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(4224ca4d93ce4784b52a7550468ce1b1): perf score=0.033356
I20240503 14:02:18.245745 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(355b5456887b4c5694badc9d53b1809a): perf score=0.033368
I20240503 14:02:18.258549 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(355b5456887b4c5694badc9d53b1809a) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1324,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":383,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":225}
I20240503 14:02:18.259349 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(0e442bb664f44887aa8afdb62e1d97fc): 4838 bytes on disk
I20240503 14:02:18.259696 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(0e442bb664f44887aa8afdb62e1d97fc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 14:02:18.260380 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(355b5456887b4c5694badc9d53b1809a): 4583 bytes on disk
I20240503 14:02:18.262712 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(355b5456887b4c5694badc9d53b1809a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 14:02:18.274194 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(4224ca4d93ce4784b52a7550468ce1b1) complete. Timing: real 0.028s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1524,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":639,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":79,"threads_started":1,"wal-append.queue_time_us":793}
I20240503 14:02:18.275130 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(4224ca4d93ce4784b52a7550468ce1b1): 4837 bytes on disk
I20240503 14:02:18.275552 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(4224ca4d93ce4784b52a7550468ce1b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240503 14:02:18.276350 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(355b5456887b4c5694badc9d53b1809a): 4583 bytes on disk
I20240503 14:02:18.278370 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(355b5456887b4c5694badc9d53b1809a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 14:02:18.279412 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:18.289916 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:18.291625 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.292955 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:18.294730 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:18.295995 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:18.311939 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.312587 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:18.330868 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:18.390981 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:18.461691 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:18.464696 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.467306 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:18.467571 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.470798 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:18.478331 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:18.487578 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:02:18.523016 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:18.587014 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:18.588227 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:18.589267 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:18.590005 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.591549 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:18.596489 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.660905 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:18.663020 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.665885 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:18.671108 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.671378 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:18.672863 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:18.687754 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:18.770615 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:18.773669 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:18.779132 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.780791 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:18.782665 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:18.782837 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:18.813864 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:18.823391 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:18.904628 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.905584 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:18.906001 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:18.906368 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:18.921051 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:18.926177 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:18.933941 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:18.942875 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:18.978945 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:18.984321 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:18.996018 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:18.996018 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.005630 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.016953 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(adf2c44328e244779955e712b99f8331): perf score=0.033379
I20240503 14:02:19.026116 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:19.031594 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(df13ec098dc548afac8c67195d25e7e8): perf score=0.033371
I20240503 14:02:19.035065 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(adf2c44328e244779955e712b99f8331) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3465,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1550,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":160,"threads_started":2,"wal-append.queue_time_us":687}
I20240503 14:02:19.036060 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(adf2c44328e244779955e712b99f8331): 4923 bytes on disk
I20240503 14:02:19.036429 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(adf2c44328e244779955e712b99f8331) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 14:02:19.047478 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(df13ec098dc548afac8c67195d25e7e8) complete. Timing: real 0.016s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1493,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":501,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":163}
I20240503 14:02:19.048346 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(df13ec098dc548afac8c67195d25e7e8): 4586 bytes on disk
I20240503 14:02:19.050220 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(df13ec098dc548afac8c67195d25e7e8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 14:02:19.108314 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.108462 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:19.109019 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:19.112562 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:19.112567 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:19.133093 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:19.149772 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.169893 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.171635 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:19.181216 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.181337 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:19.181545 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:19.204617 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:19.230257 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:19.276566 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:19.277103 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:19.282018 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.283195 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.288913 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:19.295295 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:19.324489 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:19.364985 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:19.389215 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:19.411531 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:19.494218 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:19.494503 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.495667 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:19.509244 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:19.509967 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.511723 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:19.527954 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:02:19.548117 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:02:19.554984 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:19.608420 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:19.613792 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.629884 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:19.630199 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.644793 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:19.653748 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:19.662299 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:19.672901 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:19.678601 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:19.685627 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.686197 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.686693 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:19.698499 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:02:19.707067 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:19.718622 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:19.790740 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.791771 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:19.796195 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:19.797160 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:19.798345 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.800835 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:19.893532 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:19.906493 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:19.910811 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:19.911042 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:19.913465 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.923206 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:19.930266 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:19.950595 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:19.982632 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:19.994586 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:20.019034 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:20.020061 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:20.021184 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.027647 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:20.032627 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:20.040630 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:20.051010 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:20.074600 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:20.091786 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:20.092236 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:20.092450 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:20.096211 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:20.099318 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:20.110323 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:20.120586 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.190310 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:20.196816 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:20.199391 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:20.200688 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:20.207108 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.211601 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.266589 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:20.270356 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:20.272909 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:20.277546 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:20.277799 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.291323 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(faa0a6ca00d442eaaf9fca7fc1259ec7): perf score=0.033352
I20240503 14:02:20.294131 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.310405 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(faa0a6ca00d442eaaf9fca7fc1259ec7) complete. Timing: real 0.019s user 0.012s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":567,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":137}
I20240503 14:02:20.311380 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(faa0a6ca00d442eaaf9fca7fc1259ec7): 4583 bytes on disk
I20240503 14:02:20.311918 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(faa0a6ca00d442eaaf9fca7fc1259ec7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240503 14:02:20.353216 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:20.369516 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240503 14:02:20.411288 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:20.411636 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:20.447839 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:20.462188 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:20.499671 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:20.516940 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:20.517257 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:20.519058 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.523262 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:20.525673 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:20.526391 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.544420 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:20.608278 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:20.638991 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:20.641580 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.645725 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:20.648212 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:20.655864 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:20.658800 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:20.667311 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:20.699649 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:20.700883 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:20.713303 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:20.715978 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.719413 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:20.721515 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:20.723152 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:20.751349 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:20.752079 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:20.801210 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:20.806070 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:20.824319 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:20.827152 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:20.829452 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:20.834460 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.839025 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.853598 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:20.891184 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:20.939518 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:20.943312 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:20.943595 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.951432 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:20.955693 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:20.962574 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:20.996104 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:21.084380 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:21.085968 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:21.086850 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:21.087661 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:21.089358 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:21.095268 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:21.096136 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:21.105557 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:21.108084 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:21.115334 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:21.123775 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:21.128438 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:21.142944 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:21.230955 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:02:21.245311 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:21.245499 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:21.246222 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:21.256456 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:21.262866 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:21.267692 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:21.275668 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:21.281735 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:21.283466 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:21.293587 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:21.304585 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:21.306211 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:21.316514 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:21.360718 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:21.468618 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:21.517227 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:21.593093 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:21.605721 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:21.641125 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:21.641978 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:21.646917 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:21.647594 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:21.654191 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:21.656937 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:21.668012 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:21.674460 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:21.676769 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:21.687650 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:21.691757 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:21.694533 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:21.702494 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:21.765838 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:21.779911 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:21.821159 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:21.828269 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:21.830232 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:21.832584 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:21.839411 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:21.841440 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:21.857105 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:21.866191 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:21.875677 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:21.880806 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:21.886085 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:21.896778 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:21.898579 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:21.899143 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:21.968617 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:21.993604 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:22.003933 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.003991 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:22.005115 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.006019 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:22.006743 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:22.031126 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:02:22.050917 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:22.091747 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:02:22.109059 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:22.113126 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.115370 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.117465 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:22.120340 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:22.121775 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:22.210636 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 14:02:22.232353 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:22.233503 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:22.238368 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.249035 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:22.250650 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:22.253494 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.285969 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.286329 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.288491 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:22.289628 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:22.297904 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:02:22.299940 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:22.305585 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:22.317164 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:22.345109 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:02:22.359637 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:02:22.383939 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:22.385931 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:22.386296 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:22.391288 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:22.394352 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:22.395058 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:22.414312 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.415102 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:22.552006 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:22.609207 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:22.611176 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:22.611845 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:22.617870 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:22.618371 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.620096 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:22.637704 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.650010 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:22.717821 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:22.719501 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:22.721343 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.723290 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.736734 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:22.739158 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:22.771044 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:02:22.780380 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:22.800462 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:22.800714 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:22.815953 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.816563 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.818866 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:22.825915 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:22.840304 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:22.846758 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:02:22.872048 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:22.901477 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:22.903852 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:22.904557 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:22.914942 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:22.921669 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:22.939108 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:22.969301 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:22.985880 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:22.988222 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:22.988279 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:23.002398 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:23.007529 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:23.009145 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.013475 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:23.076285 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:02:23.100502 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:23.105805 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:23.113894 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:23.114739 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.116436 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:23.117398 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:23.119642 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:23.126216 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:23.184815 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:23.227361 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.229882 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:23.232921 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.237721 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:23.241185 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:23.252528 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:23.277942 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:02:23.290952 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:23.292788 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:23.299736 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:23.299962 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.299963 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.305207 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:23.311533 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:23.417650 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:23.419685 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:23.422075 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:23.424974 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.425418 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:23.438365 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:23.479051 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:23.550172 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:23.563367 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:23.566834 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.568156 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:23.569830 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.571038 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:23.574452 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:23.578132 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:23.591771 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:23.695816 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:23.701021 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.701244 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:23.704926 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.706173 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:23.715647 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:23.753470 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:23.776955 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:23.781993 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:23.783171 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.785504 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:23.786370 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:23.788224 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:23.865492 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:02:23.873793 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.880432 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:23.883433 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:23.884105 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:23.886729 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.893055 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:23.907457 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:23.956123 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:23.956468 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.962917 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:23.968114 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:23.968523 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:23.973307 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:24.011531 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:02:24.011659 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:24.051481 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:24.054360 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:24.056902 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:24.063427 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.066368 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:24.067411 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:24.089560 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:24.149005 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:24.164158 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:24.164217 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:24.172974 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.184054 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.184649 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:24.191617 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:24.233109 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:24.284994 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:24.285207 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:24.286679 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.298323 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:24.305565 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.313469 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:24.316260 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:24.404680 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:24.409245 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:24.412407 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:24.415153 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:24.419713 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.425153 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:24.472977 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:02:24.498749 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:24.506320 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:24.507359 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:24.510052 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.511708 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.525462 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:24.526082 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:24.550424 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:02:24.617528 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:24.618220 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:24.619189 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.624269 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.631662 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:24.633920 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:24.658879 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:24.739879 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:24.754179 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:24.759626 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:24.761824 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:24.764307 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.768504 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:24.772848 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.787930 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:24.788208 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:24.831296 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:24.833441 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:24.835114 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:24.837474 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:24.841980 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:24.844300 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.865869 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:24.880667 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:02:24.928390 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:02:24.931494 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:24.959901 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:24.962383 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:24.964624 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.964774 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:24.965190 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:24.965373 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:24.981611 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:24.993024 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:25.025177 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:25.025800 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:25.028113 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.028573 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:25.032060 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:25.033008 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:25.041703 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:25.053356 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:25.140048 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:25.140410 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.140476 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:25.140902 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:25.141121 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:25.143491 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:25.179514 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:02:25.197687 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:25.307436 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:25.312851 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.312902 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:25.314108 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:25.328434 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.337570 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:25.385418 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 165 similar messages]
W20240503 14:02:25.388682 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:25.434114 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:25.438310 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:25.444056 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:25.450732 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:25.453723 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.459720 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:25.496945 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:25.507149 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:25.507409 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:25.509253 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.515740 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.518270 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:25.528774 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:25.529600 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:25.546169 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:02:25.551699 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:25.575762 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:25.579485 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(92506cde7b084b14b88d4a915731eeb3): perf score=0.033354
I20240503 14:02:25.594349 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(92506cde7b084b14b88d4a915731eeb3) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":143,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1917,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":570,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":143}
I20240503 14:02:25.595521 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(92506cde7b084b14b88d4a915731eeb3): 4583 bytes on disk
I20240503 14:02:25.596478 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(92506cde7b084b14b88d4a915731eeb3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 14:02:25.625160 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:25.625864 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:25.626585 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.635421 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:25.639382 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.642246 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:25.668112 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:25.700546 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:02:25.718919 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:02:25.733044 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:25.743028 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:25.747735 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.753412 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.757118 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:25.763883 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:25.767541 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:25.828367 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:25.831708 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:25.837488 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.854579 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.856115 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:25.856395 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:25.884794 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:25.923928 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:25.927309 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:25.927709 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.928292 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:25.929239 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:25.946672 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:26.003155 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:26.012918 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:26.015487 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.018770 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.023038 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:26.023788 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:26.029562 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:26.062702 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:02:26.067349 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:26.096490 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:26.097056 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.097313 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:26.105074 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:26.105419 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.106597 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:26.143584 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:26.249248 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 14:02:26.323119 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(a6a89230883347d4a64d96d321247012): perf score=0.033356
I20240503 14:02:26.330343 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.338629 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:26.341663 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.342445 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(a6a89230883347d4a64d96d321247012) complete. Timing: real 0.019s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":122,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1476,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":120,"threads_started":2,"wal-append.queue_time_us":576}
I20240503 14:02:26.343014 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:26.343566 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(a6a89230883347d4a64d96d321247012): 4838 bytes on disk
I20240503 14:02:26.344766 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(a6a89230883347d4a64d96d321247012) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 14:02:26.351493 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:26.364192 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:26.397316 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:02:26.403002 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:26.446209 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:26.447420 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.448321 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:26.451174 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:26.452100 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.452447 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:26.499712 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:26.514906 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:26.515676 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.517982 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.520853 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:26.523134 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:26.542889 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:26.594909 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:26.595695 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:26.596302 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.603945 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:26.607791 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:26.611953 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:26.618939 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:26.628814 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:26.654116 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:26.696790 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:26.713265 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:26.713286 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:26.716569 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:26.721086 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:26.725406 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.726102 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:26.747526 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:26.842535 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:26.849704 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:26.850854 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:26.855510 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:26.859417 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.880790 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:26.885915 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.889461 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:26.896041 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:26.909715 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:26.909760 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:26.910660 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:26.921792 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:26.950145 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:27.004791 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:27.030637 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:27.035009 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:27.038642 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:27.039181 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.039479 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.044333 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:27.052357 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:27.059604 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:27.067509 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.076623 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:27.079682 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:27.089959 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:27.101665 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:27.204322 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:27.240336 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:27.277292 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:27.298952 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 14:02:27.351904 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(e92d8f55591346158b9583feb958a8ca): perf score=0.033395
I20240503 14:02:27.353822 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(5e6ca408c4254678b67a74f49a34881b): perf score=0.033397
W20240503 14:02:27.367223 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:27.371003 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:27.377373 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(5e6ca408c4254678b67a74f49a34881b) complete. Timing: real 0.023s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4034,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1818,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":567,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":510,"threads_started":2,"wal-append.queue_time_us":132}
I20240503 14:02:27.378369 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(5e6ca408c4254678b67a74f49a34881b): 4583 bytes on disk
I20240503 14:02:27.379411 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(5e6ca408c4254678b67a74f49a34881b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240503 14:02:27.388998 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(e92d8f55591346158b9583feb958a8ca) complete. Timing: real 0.037s user 0.013s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":5731,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1472,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":705,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":4011}
I20240503 14:02:27.389998 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(e92d8f55591346158b9583feb958a8ca): 4922 bytes on disk
I20240503 14:02:27.392398 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(e92d8f55591346158b9583feb958a8ca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240503 14:02:27.406642 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:27.409636 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:27.411602 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:27.413595 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.417752 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:27.431388 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:27.434398 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.448854 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.468232 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.474139 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:27.475181 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:02:27.476624 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:27.477753 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:27.489560 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:27.491425 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:27.502846 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:27.510267 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:27.535869 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:27.590580 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:27.593475 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:27.599485 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:27.613164 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:27.614732 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:27.623893 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.627528 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:27.629696 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:02:27.642428 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.659462 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.659715 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:27.662429 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:27.667660 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:27.751106 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:27.755479 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.758396 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:27.760828 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.767210 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:27.771782 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:27.787703 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:27.799779 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:27.826941 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:27.830365 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:27.844204 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:27.874686 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:27.877660 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:27.877923 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:27.887250 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:27.887315 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:27.888906 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:27.918327 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:28.005851 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.007311 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:28.008352 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:28.010424 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:28.013996 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:28.024209 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.024878 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:28.030781 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:28.044752 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:28.046710 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.053999 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:28.055603 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:28.069813 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:28.073643 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:28.137413 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(70ffc5888a5945d6a8e166420e6f1bd9): perf score=0.033385
I20240503 14:02:28.156639 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(70ffc5888a5945d6a8e166420e6f1bd9) complete. Timing: real 0.019s user 0.012s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":306,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1635,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":577,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":1607}
I20240503 14:02:28.157728 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(70ffc5888a5945d6a8e166420e6f1bd9): 4838 bytes on disk
I20240503 14:02:28.158315 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(70ffc5888a5945d6a8e166420e6f1bd9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240503 14:02:28.167804 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:28.168403 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:28.179890 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.182168 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.183225 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:28.198884 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:28.247301 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(92506cde7b084b14b88d4a915731eeb3): perf score=0.033391
W20240503 14:02:28.247561 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:28.258769 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(92506cde7b084b14b88d4a915731eeb3) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":175,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1534,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":406,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":117}
I20240503 14:02:28.259713 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(92506cde7b084b14b88d4a915731eeb3): 4583 bytes on disk
I20240503 14:02:28.260011 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(92506cde7b084b14b88d4a915731eeb3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240503 14:02:28.306167 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:28.356014 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:28.357827 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:28.368186 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.370244 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:28.374521 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.375804 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:28.378804 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:28.417526 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling FlushMRSOp(a0106f7a40cb467eb4738890ac4e22d4): perf score=0.033387
I20240503 14:02:28.434239 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: FlushMRSOp(a0106f7a40cb467eb4738890ac4e22d4) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":1511,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":559,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":70,"threads_started":1,"wal-append.queue_time_us":149}
I20240503 14:02:28.435082 13953 maintenance_manager.cc:419] P b67f337e10994234baedbf0b19c13513: Scheduling UndoDeltaBlockGCOp(a0106f7a40cb467eb4738890ac4e22d4): 4923 bytes on disk
I20240503 14:02:28.435578 13843 maintenance_manager.cc:643] P b67f337e10994234baedbf0b19c13513: UndoDeltaBlockGCOp(a0106f7a40cb467eb4738890ac4e22d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240503 14:02:28.436542 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:28.439476 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:28.496899 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:28.498543 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:28.500349 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:28.500445 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:28.507645 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:28.510133 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:28.510217 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:28.511677 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.517292 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.542187 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:28.549664 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:28.550261 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.565379 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.571969 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(3481715d755a4277b0905b0832fad4af): perf score=0.033345
W20240503 14:02:28.586808 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:28.592741 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(3481715d755a4277b0905b0832fad4af) complete. Timing: real 0.021s user 0.015s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6544,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":563,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":289}
I20240503 14:02:28.593700 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(3481715d755a4277b0905b0832fad4af): 4838 bytes on disk
I20240503 14:02:28.594152 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(3481715d755a4277b0905b0832fad4af) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240503 14:02:28.650437 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling FlushMRSOp(f4f9cb87c70e43dbb0d850c275ebb8a5): perf score=0.033380
I20240503 14:02:28.666999 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: FlushMRSOp(f4f9cb87c70e43dbb0d850c275ebb8a5) complete. Timing: real 0.016s user 0.011s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3357,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":662,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":292}
I20240503 14:02:28.667960 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(f4f9cb87c70e43dbb0d850c275ebb8a5): 4836 bytes on disk
I20240503 14:02:28.668428 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(f4f9cb87c70e43dbb0d850c275ebb8a5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 14:02:28.680181 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:28.685552 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.686523 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.695057 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:28.697527 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:28.707579 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:28.711401 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.718217 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:28.719686 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:28.720816 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.740097 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:28.762089 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:28.764511 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:28.843178 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:28.843636 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:28.844307 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:28.844774 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.845378 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:28.857004 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:28.861744 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:28.881417 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:28.913110 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:28.947520 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:28.949946 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:28.950342 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:28.952656 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:28.953617 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:28.956977 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:28.967116 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:29.008642 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:29.029645 5540 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:02:29.029767 5540 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:02:29.030303 5540 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:02:29.030753 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:02:29.030871 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:02:29.030941 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:02:29.032181 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:29.033694 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:29.033778 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:02:29.034405 5540 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:02:29.051316 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:29.051401 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:29.057762 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:29.058673 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.060230 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:29.072175 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:29.078447 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.158876 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:29.160976 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:29.161998 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:29.162292 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:29.164168 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.180490 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.187522 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:29.234659 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:02:29.251750 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:02:29.264176 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:29.265491 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:29.265836 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.267537 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:29.269454 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:29.275573 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.347210 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling FlushMRSOp(5e6ca408c4254678b67a74f49a34881b): perf score=0.033362
I20240503 14:02:29.363030 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: FlushMRSOp(5e6ca408c4254678b67a74f49a34881b) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":404,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":2537,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":596,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":127}
I20240503 14:02:29.363716 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(5e6ca408c4254678b67a74f49a34881b): 4583 bytes on disk
I20240503 14:02:29.364050 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(5e6ca408c4254678b67a74f49a34881b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240503 14:02:29.382000 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:29.382449 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.383469 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:29.387079 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:29.388839 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:29.390002 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.412271 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:29.431692 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:29.488132 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:29.496026 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:29.496990 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.503881 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:29.507087 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.507735 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:29.552342 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:29.555711 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.557534 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.561012 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:29.562443 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:29.572976 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:29.579070 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:02:29.586041 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:29.662636 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.663393 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.663841 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:29.664062 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:29.670940 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:29.672269 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:29.692456 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:29.708483 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(5e6ca408c4254678b67a74f49a34881b): perf score=0.033394
I20240503 14:02:29.718186 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:29.721542 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:29.728331 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.730501 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:29.735656 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(5e6ca408c4254678b67a74f49a34881b) complete. Timing: real 0.027s user 0.014s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1486,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":111,"threads_started":2,"wal-append.queue_time_us":111}
I20240503 14:02:29.735976 5604 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:02:29.736071 5604 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:02:29.736415 5604 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
I20240503 14:02:29.737444 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(5e6ca408c4254678b67a74f49a34881b): 4583 bytes on disk
W20240503 14:02:29.738057 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:29.740375 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:29.740432 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:02:29.740967 5604 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:02:29.742573 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(5e6ca408c4254678b67a74f49a34881b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 14:02:29.743279 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling FlushMRSOp(2089eeda1e0f4ff5926584d472e6631d): perf score=0.033345
I20240503 14:02:29.747743 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:29.749522 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.764930 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(faa0a6ca00d442eaaf9fca7fc1259ec7): perf score=0.033402
I20240503 14:02:29.767817 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: FlushMRSOp(2089eeda1e0f4ff5926584d472e6631d) complete. Timing: real 0.024s user 0.012s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6068,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1282,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":551,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":143}
I20240503 14:02:29.768536 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(2089eeda1e0f4ff5926584d472e6631d): 4923 bytes on disk
I20240503 14:02:29.768923 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(2089eeda1e0f4ff5926584d472e6631d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 14:02:29.778640 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(faa0a6ca00d442eaaf9fca7fc1259ec7) complete. Timing: real 0.014s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2935,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1382,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":512,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":205}
I20240503 14:02:29.779492 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(faa0a6ca00d442eaaf9fca7fc1259ec7): 4583 bytes on disk
I20240503 14:02:29.779817 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(faa0a6ca00d442eaaf9fca7fc1259ec7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 14:02:29.780647 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(c42fee9fadbb4601a7fe32eb6a04e767): perf score=0.033374
I20240503 14:02:29.794658 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(c42fee9fadbb4601a7fe32eb6a04e767) complete. Timing: real 0.014s user 0.009s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1227,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":443,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":139}
I20240503 14:02:29.795547 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(c42fee9fadbb4601a7fe32eb6a04e767): 4923 bytes on disk
I20240503 14:02:29.795861 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(c42fee9fadbb4601a7fe32eb6a04e767) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
W20240503 14:02:29.839489 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:29.845024 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:29.845034 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:29.847642 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:29.855048 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.855208 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.857363 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:29.859781 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:29.871934 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:02:29.875121 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:02:29.922230 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:29.971963 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:29.984766 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:29.988503 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:29.991111 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:29.994668 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:29.994877 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:30.007783 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.047102 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(a6a89230883347d4a64d96d321247012): perf score=0.033370
W20240503 14:02:30.048238 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:30.059945 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(a6a89230883347d4a64d96d321247012) complete. Timing: real 0.013s user 0.011s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1432,"drs_written":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4,"lbm_write_time_us":489,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":127,"threads_started":1,"wal-append.queue_time_us":205}
I20240503 14:02:30.060678 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(a6a89230883347d4a64d96d321247012): 4838 bytes on disk
I20240503 14:02:30.060984 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(a6a89230883347d4a64d96d321247012) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240503 14:02:30.098681 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.100447 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:30.101972 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.107952 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:30.109170 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:30.119208 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:30.130018 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:30.132123 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:02:30.139859 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:30.189337 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:30.194937 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:30.195019 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:30.196091 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.205772 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:30.206318 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:30.207917 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:30.237689 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:30.237990 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:30.293989 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.294238 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.295378 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:30.296841 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:30.312320 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling FlushMRSOp(c2e9f6eecbec45988b290775277dbddc): perf score=0.033392
I20240503 14:02:30.316848 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:30.317239 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:30.328459 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: FlushMRSOp(c2e9f6eecbec45988b290775277dbddc) complete. Timing: real 0.016s user 0.006s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1517,"drs_written":1,"lbm_read_time_us":31,"lbm_reads_lt_1ms":4,"lbm_write_time_us":583,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":163}
I20240503 14:02:30.329216 12477 maintenance_manager.cc:419] P 84c63c6f551949f6a1d7c07760824361: Scheduling UndoDeltaBlockGCOp(c2e9f6eecbec45988b290775277dbddc): 4583 bytes on disk
I20240503 14:02:30.329512 12367 maintenance_manager.cc:643] P 84c63c6f551949f6a1d7c07760824361: UndoDeltaBlockGCOp(c2e9f6eecbec45988b290775277dbddc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240503 14:02:30.354583 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:02:30.390872 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 156 similar messages]
W20240503 14:02:30.425244 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:30.430142 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:30.437512 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:30.440052 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:30.440760 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.443886 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:30.445506 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:30.459908 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:30.460142 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:30.468199 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:30.468811 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:30.489684 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:30.490161 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:30.490942 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.491649 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:30.577211 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:30.582234 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:30.582677 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:30.596151 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.605670 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:30.606110 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:30.610649 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:30.611796 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:30.626941 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.630162 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:30.632566 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.650202 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:30.662549 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:30.739166 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:30.739840 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:30.741230 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:30.746224 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:30.746902 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.755398 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:30.764000 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:30.877960 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:02:30.920287 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:30.942023 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:30.944804 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:30.945145 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:30.945786 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.947587 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:30.971792 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:30.992321 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:02:31.004467 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:31.063833 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:31.071695 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.072835 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:31.074658 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:31.076963 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:31.082028 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.096168 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.097312 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:31.099749 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:31.110437 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:31.113495 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.114377 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:31.142940 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:02:31.206861 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:31.210976 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:31.211337 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.213055 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:31.214891 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:31.230300 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:31.231227 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.231372 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:31.283270 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:31.308895 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:31.312717 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:31.313725 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.314528 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:31.323846 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:31.324491 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:31.379364 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:02:31.421696 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:31.430850 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:31.437000 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:31.438719 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:31.440017 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:31.441985 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:31.447670 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.456135 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:31.480376 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:31.491160 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:31.492476 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:31.500150 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.501585 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:31.518283 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:31.527026 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:31.533174 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.585773 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:31.587064 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:31.588470 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.591740 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.595481 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:31.598802 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:31.659817 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:02:31.661733 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.661733 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:31.664652 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:31.670727 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:31.671835 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:31.673687 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:02:31.678517 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:31.698601 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:31.725694 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:31.837611 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:31.853379 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:31.901463 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:31.902233 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:31.907124 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:31.912528 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:31.915117 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:31.917387 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:31.917657 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:32.010454 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:02:32.022600 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:32.028175 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.033166 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.034546 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:32.035213 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.040475 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:32.051872 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:32.053689 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:32.054123 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:32.059736 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:32.061791 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:32.086915 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:02:32.101716 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.187160 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:32.188292 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:32.196203 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.200554 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:32.203402 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:32.216981 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:32.290510 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:32.301882 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.303320 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:32.305470 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:32.306229 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:32.306897 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:32.314286 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:32.326874 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:32.353757 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:32.357290 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:32.361023 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:32.403914 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:32.404093 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:32.414709 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:32.414709 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:32.415052 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:32.417735 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.429867 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:32.435402 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:32.457211 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:02:32.459097 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:32.485983 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:32.491001 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:32.491729 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:32.493842 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:32.499738 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:02:32.500504 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:32.500634 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:02:32.518982 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:32.575481 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:32.578094 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:32.581475 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.585129 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:32.587899 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:32.598559 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.654414 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:32.655267 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.659181 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:32.661988 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:32.663097 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:32.663385 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:32.732959 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:32.736740 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:32.775825 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:02:32.785562 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:32.827644 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:32.859117 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:32.880240 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:32.890626 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:32.891172 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:32.892731 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.893949 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:32.897958 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:32.899526 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:32.995414 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.000604 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:33.001171 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:33.008731 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.008826 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:33.021242 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:33.046700 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:02:33.066834 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.072995 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:33.073648 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:33.076478 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:33.077356 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:33.080163 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:02:33.103966 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:33.154978 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:33.159097 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.159097 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:33.171800 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.180193 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:33.189770 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:33.270141 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.271940 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.272372 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:33.272927 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:33.298563 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:33.298787 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:33.299542 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:33.340535 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:33.375720 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:33.392635 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:33.407653 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:33.408418 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:33.410003 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:33.413235 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.419126 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:33.419632 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:33.421152 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.435514 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:33.437368 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:33.446059 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:33.447661 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:33.466374 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.471518 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:33.553148 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:33.555464 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:33.560478 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:33.560726 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.571792 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:33.579557 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.593614 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:33.596114 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:33.621794 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.628080 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:33.628588 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.632211 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:33.702039 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:33.718515 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:33.726712 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:33.735919 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:33.802901 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:33.821655 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:33.875788 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:02:33.898068 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:33.898875 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:33.900450 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:33.903622 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.910020 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:33.912271 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.921041 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:33.962936 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:33.966032 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:33.966259 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:33.966431 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:33.978569 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:33.984161 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:33.993086 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:34.020134 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:34.067874 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.070677 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:34.071242 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:02:34.072396 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.073540 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:34.076772 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:34.089771 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:34.129783 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.131693 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:34.134133 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:34.152084 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.153857 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:34.154592 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:34.179672 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:34.224926 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:34.227581 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:34.228415 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.233067 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.240134 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:34.247629 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:34.282691 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:34.328060 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:34.328717 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.329340 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:34.329758 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:34.334331 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.344264 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:34.380981 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:34.403043 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:34.440683 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:34.442632 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.446406 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:34.451021 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:34.452004 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:34.452565 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:34.483882 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:34.502923 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:34.503330 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:34.507066 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.512002 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.516290 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:34.518038 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:34.543337 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:34.605857 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.608211 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:34.610703 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:34.612339 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:34.620817 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:34.644685 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:34.669291 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:34.704135 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:34.723351 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:34.760205 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:34.765699 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:34.765866 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.769152 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.777045 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:34.780824 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:34.784575 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:34.812285 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:34.826076 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:34.901237 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:34.906085 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:34.908640 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:34.910656 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:34.912704 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:34.915340 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.920158 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:34.942358 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:34.950301 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:34.976819 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:34.981225 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:34.990609 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:34.993505 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.003808 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:35.017618 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:35.070003 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.071400 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:35.073377 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:35.073662 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:35.074450 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:35.094954 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:35.140646 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:02:35.148725 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:35.149070 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:35.154142 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:35.165992 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.167382 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.167917 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:35.209481 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:35.214427 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:35.233520 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:35.234885 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:35.238286 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.248654 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:35.251152 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.255671 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:35.269843 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:35.334789 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:35.338029 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:35.339357 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:35.339949 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:35.342223 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:35.356381 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.361377 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:35.362274 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:35.404419 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 163 similar messages]
I20240503 14:02:35.461321 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.462334 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:35.470152 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:35.470870 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:35.479096 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.487694 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:35.556840 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:35.572108 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:35.572556 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.574822 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:35.574872 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:35.577292 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:35.584559 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:35.587255 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:35.674332 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:35.675846 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:35.676541 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.690234 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:35.693066 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:35.694628 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:35.705660 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:35.724865 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:35.780539 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:35.781944 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:35.786810 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:35.787278 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.789754 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.790207 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:35.806305 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:35.868355 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:35.885838 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:35.909499 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:35.910503 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.916013 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:35.919373 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:35.922560 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:35.928918 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:35.929360 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.932160 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:35.932572 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:35.941193 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:35.944186 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:35.968977 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:35.971360 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:36.075042 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:36.082770 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:36.084074 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:36.091647 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:36.097157 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.103449 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:36.105906 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:36.107092 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.107753 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:36.127380 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:36.127691 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:36.134372 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.141275 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:36.166936 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:36.201763 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:36.229995 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:36.239619 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.245365 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:36.246001 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.251421 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:36.252441 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:36.255074 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:36.263788 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:36.333029 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:36.334903 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:02:36.380623 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:02:36.428432 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:36.435688 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:36.436273 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.436594 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.437832 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:36.451656 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:36.457471 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:36.541729 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:36.550290 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.553835 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:36.554699 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.555660 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:36.579185 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:36.582551 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:36.582793 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.591796 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.595387 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:36.606477 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:36.620689 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:36.647053 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 14:02:36.664294 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:36.705193 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:36.707595 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:36.709142 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:36.709760 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:36.712286 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:36.713196 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.722476 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:36.735494 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:36.817462 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:36.821101 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.821429 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.821990 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:36.823621 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:36.846124 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:02:36.846470 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:36.847147 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:36.880137 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:36.914538 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:36.917019 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:36.923178 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:36.927634 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:02:36.928745 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:36.931644 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.932438 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.942891 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:36.965376 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:36.965833 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:36.968945 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:36.982698 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:36.989763 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.002199 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:37.081339 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:37.097771 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:37.106251 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:37.112401 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:37.113976 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.116802 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:37.117390 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:37.120790 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:37.121022 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.123337 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.124362 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:37.125366 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:37.155678 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:37.160173 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.163360 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:37.226752 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:02:37.280551 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:37.282158 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:37.314424 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:37.354054 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:37.452252 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:37.455965 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:37.457880 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.473567 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.474886 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:37.491385 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:37.505906 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:37.507685 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:37.509399 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:02:37.509511 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:37.515702 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:37.532497 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:37.533097 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:37.580875 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:02:37.609923 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 14:02:37.617662 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:37.620487 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.627774 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:37.631526 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.640401 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:37.645644 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:37.655655 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:37.658335 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:37.663713 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.664291 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:37.667125 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:37.681500 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:37.689205 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:37.726451 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:37.783488 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:37.784968 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:37.787345 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.787345 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.787966 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:37.793483 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:37.836624 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:37.847884 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:02:37.853032 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:37.888633 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:37.892035 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.895407 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:37.898504 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:37.903401 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:37.903879 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:38.004598 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:38.012938 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:38.016305 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:38.018028 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.020725 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.027712 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.037104 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:38.040644 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:38.049626 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:38.050104 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:38.058229 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:38.076902 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:38.077661 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:02:38.081195 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:38.115440 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:38.141353 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.142493 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.144905 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:38.145529 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:38.160034 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:38.162488 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:38.236093 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:38.284021 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:02:38.297927 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:02:38.356240 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:38.364130 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:38.365775 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:38.366552 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:38.367748 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:38.370783 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:38.375602 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:38.376525 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.380342 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:38.478770 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:38.481719 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:38.482138 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.495398 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:38.497783 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.498236 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:38.521004 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:38.521584 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:38.534268 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:38.535349 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:38.536865 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.539635 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:38.618363 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:38.637748 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.638077 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.638087 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:38.643486 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:38.655591 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:38.665747 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:38.667157 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:38.682451 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:38.692910 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:38.694902 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:38.698890 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:38.703562 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.710315 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.724202 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:38.783521 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:38.783905 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.785478 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:38.786077 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.787264 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:38.797328 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:38.809733 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:38.891513 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:38.892443 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:38.892465 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:38.900816 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:38.900859 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:38.900985 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:38.903857 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:38.914778 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:02:38.923972 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:38.965283 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:39.013511 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:39.016284 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:39.017508 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.019923 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:39.023736 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:39.044313 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:39.058444 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:02:39.089051 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:39.120100 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.121652 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:39.122958 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:39.125658 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.134645 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:39.135941 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:39.212675 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:39.213989 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:39.214280 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.217881 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.219974 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:39.227921 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:39.231666 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:39.234696 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:39.274111 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:02:39.294626 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:39.319432 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.320746 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:39.325093 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:39.326117 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:39.329804 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.335522 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:39.357156 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:02:39.407999 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:39.447566 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.447790 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:39.456851 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:39.456941 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:39.458974 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:39.470669 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:39.472443 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.473196 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:39.478861 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.481856 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:39.492980 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:39.503755 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:39.517760 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:39.611809 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:39.613868 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:39.614495 12016 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:39.615355 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:39.618433 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:39.621493 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:39.624894 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:39.626506 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.636395 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.649359 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:39.652021 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:39.655885 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:39.659302 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:39.663545 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:39.670881 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:39.710426 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:39.755427 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:39.757501 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:39.760654 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.762362 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.770044 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:39.780568 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:39.814409 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:39.837414 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:39.855907 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:02:39.856357 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:39.961397 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:39.964186 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.967595 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:39.970883 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:39.971410 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:39.984804 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:39.993371 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:40.077486 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:40.078290 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:40.081248 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.089440 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.091843 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:40.096760 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:40.110260 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:40.137308 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:40.139047 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.139907 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:40.145923 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:40.147964 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.149123 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:40.236394 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.236498 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.236754 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:40.237236 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:40.247579 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:40.252250 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:40.292937 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:02:40.324784 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:40.326038 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:40.326969 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:40.334704 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:40.337649 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.339753 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:40.341156 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.351773 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:40.363677 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:40.413467 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240503 14:02:40.420028 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:40.451122 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:40.456497 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.457294 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:40.458369 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:40.458845 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:40.463008 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.513396 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.513707 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:40.517376 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.518394 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:40.525678 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:02:40.526103 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:40.528883 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:40.541745 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:40.543861 8888 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:02:40.544018 8888 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:02:40.544447 8888 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:02:40.548612 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:02:40.548746 12036 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:02:40.548820 12036 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:02:40.550508 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:40.550637 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:40.550693 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:02:40.550824 8888 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:02:40.560781 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:40.610838 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:40.612682 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.620198 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:40.622365 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:40.622591 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:40.645646 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.682369 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:40.682554 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:40.683429 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.686806 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:40.693456 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:40.711344 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:40.751387 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:40.845760 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:40.848012 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:02:40.869513 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:02:40.921594 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:40.929162 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:40.933506 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:40.937350 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.938897 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:40.939013 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:40.946897 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:41.039327 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:41.042275 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:41.049986 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.053500 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:41.059715 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:41.085495 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:41.089268 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:41.089268 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.091992 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:41.099856 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:41.102243 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.103659 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:41.129127 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:41.144060 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:41.191188 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:41.192899 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.195830 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:41.200531 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:41.200911 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.213975 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:41.292215 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.293351 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:41.294018 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.297461 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:41.299501 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:41.305641 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:41.383774 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:41.410399 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:41.414544 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:41.415356 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:41.415720 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:41.436663 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:41.447993 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:41.450739 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.462627 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:41.465251 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.466251 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:41.466625 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:41.470319 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:41.474941 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:41.486421 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:41.516716 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:41.546758 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:41.560894 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:41.571018 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.571673 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:41.575716 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:41.576997 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:41.579479 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:41.590593 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:41.618163 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:02:41.628926 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:02:41.643816 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:41.648346 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:41.648804 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.649570 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:41.649968 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:41.651854 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:41.699290 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:02:41.773581 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:02:41.853255 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:41.856638 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:02:41.871830 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:41.894730 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:41.898186 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:41.898391 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:41.906095 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:41.911093 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:41.916270 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:41.920527 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:41.920727 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:41.936771 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:42.008111 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:42.008853 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:42.010373 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:42.010646 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.012365 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:42.019591 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:42.030174 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:42.061681 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:42.068735 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:42.069065 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.073948 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:42.079378 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.087548 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:42.094764 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:42.100283 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:02:42.154692 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:42.165740 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.166290 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:42.167292 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.177119 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:42.199914 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:42.199949 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:42.203353 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:42.222146 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:42.224504 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:42.228322 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.230953 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:42.232002 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:42.250854 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:42.251219 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.334970 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.336995 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:42.337893 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:42.338968 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:42.339448 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:42.360729 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:42.411520 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:42.440404 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:02:42.446650 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:42.448690 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:42.450171 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:42.450266 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.451584 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:42.460350 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.462749 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:42.546365 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:42.546360 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:42.547643 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:42.548656 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.563161 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:42.568971 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.569192 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:42.628202 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:02:42.653030 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:42.656425 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.657838 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:42.658831 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.666857 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:42.667471 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:42.684521 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:42.768612 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:42.771379 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:42.773722 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.773761 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:42.785620 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.791083 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:42.791285 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:02:42.887013 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:42.887024 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:42.887640 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:42.888058 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:42.892108 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:42.904012 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:42.904251 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:42.905421 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:02:42.932339 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:42.990962 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:02:42.991494 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:42.992092 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:42.992745 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:42.999154 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:43.000445 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:43.007440 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:43.011191 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:43.017726 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.028795 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:43.071262 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:43.071732 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.079695 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:43.085867 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.093631 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:43.107538 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:43.155563 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:43.161383 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.164117 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:43.171403 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:43.173705 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.182394 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:43.210214 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:02:43.224889 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:43.240810 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.242022 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:43.243886 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:43.244102 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.244756 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:43.251691 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:43.344703 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:43.346349 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:43.347049 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.347558 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:43.354717 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.362747 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:43.427692 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:02:43.450369 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:43.453361 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:43.474962 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:43.480587 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.480923 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:43.483834 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:43.491189 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:43.498544 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.576987 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:43.577302 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:43.588176 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.588385 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.589182 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:43.589272 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:43.596168 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:43.623924 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:43.678153 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:43.678323 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.679143 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.679428 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:43.680019 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:43.680773 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:43.786084 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:43.788249 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.790347 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.791590 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:43.811290 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:43.816061 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:43.919693 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:43.926824 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.934370 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:43.934998 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.941143 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:43.943332 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:43.943960 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:43.950438 8888 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:02:43.950546 8888 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:02:43.950924 8888 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
W20240503 14:02:43.951823 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:43.951954 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:43.952025 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:02:43.952168 8888 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:02:43.955370 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:02:43.955468 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:02:43.955518 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
I20240503 14:02:43.955685 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:43.957576 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:43.963375 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:43.984424 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:43.988433 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:02:43.996784 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:44.003352 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:02:44.004526 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:44.037665 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:44.085296 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:44.092314 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.094705 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:44.095525 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.097904 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:44.101263 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:44.121526 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:44.123903 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:44.124943 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:44.134650 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:44.135829 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:44.136003 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:44.143716 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:44.161338 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:44.202201 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:44.213435 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:02:44.237154 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:44.237311 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:44.240447 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.240634 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:44.253275 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.253966 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:44.325273 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:44.334123 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:02:44.399523 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:44.422039 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:02:44.451817 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.452105 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.456219 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:44.457801 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:44.467578 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:44.467803 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:44.484253 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:44.527630 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:44.532121 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:44.534197 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:44.551173 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:44.558431 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.559406 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:44.563431 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.567359 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:44.568548 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:44.588793 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:02:44.608057 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:44.612068 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:44.619318 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:44.622824 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.624727 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:44.629379 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:44.634303 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:44.643173 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:44.643265 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:44.699421 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:02:44.711489 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.711853 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:44.713557 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:44.719424 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:44.734320 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:44.735610 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.822472 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:44.827998 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:44.829550 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:44.832350 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.836221 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.855158 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:44.899436 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:44.955161 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:44.964543 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.966321 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:44.971513 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:44.976558 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:44.978008 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:44.982551 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:44.989483 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:02:44.993441 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.013360 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:45.015380 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:45.016561 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.018869 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:45.019011 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:45.075821 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:45.136348 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:45.138368 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.144520 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:45.145195 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:45.148177 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:02:45.148465 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.149008 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:45.155206 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:45.166280 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.179883 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:45.180591 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.193126 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:45.193945 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:45.204185 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:45.268793 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:45.387692 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:45.429298 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:45.461113 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:45.477972 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:45.479682 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:45.480775 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 161 similar messages]
I20240503 14:02:45.487568 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:45.488961 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.503405 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:45.515486 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:45.532533 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.533910 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:45.536345 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.545361 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:45.547312 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:45.555447 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:45.575981 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:45.587579 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:02:45.639396 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:45.643949 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:02:45.665458 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:45.667440 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.671638 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:45.674945 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:45.680647 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:45.701100 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.713243 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:45.717188 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:45.717331 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.718271 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.728714 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:45.737172 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:45.828603 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:45.829263 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:45.832197 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:45.841095 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.842644 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:45.844009 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.939937 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:45.940809 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:45.942190 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:45.948935 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:45.952386 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:45.973224 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:45.985023 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:46.002034 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:46.047567 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:46.047621 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:46.054502 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.058653 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:46.065501 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:46.068512 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:46.076423 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.078213 12672 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:02:46.078310 12672 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:02:46.078657 12672 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:02:46.080626 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:46.081137 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:46.081202 12103 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:02:46.081455 12672 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:02:46.120085 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:46.123157 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.128654 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.129120 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:46.131513 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:46.141451 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:46.150141 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:46.174794 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:02:46.220155 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:46.220638 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:46.222879 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.223074 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:46.240370 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:46.241725 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:46.252496 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:46.420228 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:46.436144 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:02:46.436800 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.440821 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:46.441885 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:46.442298 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.445052 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:46.445784 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:46.465083 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:46.489857 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:46.545430 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.551012 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:46.551211 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.552985 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:46.553069 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:02:46.556659 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:46.573139 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:46.574035 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:46.614284 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:46.617852 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.621891 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:46.623737 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:46.624001 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.638875 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:46.697590 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:02:46.721964 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.723580 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:46.726392 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.727965 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:46.733137 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:46.737671 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:46.769222 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:46.783126 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:46.793531 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:46.795339 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.795586 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:46.798727 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:46.813277 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.827484 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:46.865679 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:02:46.866330 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:46.912823 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.913738 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:46.914777 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:46.915331 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:46.916354 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:46.921638 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:46.978008 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:46.982864 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:46.996206 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:47.034096 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:47.039446 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.039463 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.039553 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:47.043125 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:47.055322 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:47.101130 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:47.103340 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.103776 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.105865 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:47.113935 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:47.114434 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:47.117987 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:47.124531 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:47.194844 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:47.209863 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:47.211872 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.215401 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:47.215708 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.216215 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:47.227514 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:47.238919 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:47.243892 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:47.285269 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:47.346911 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:47.347622 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.351486 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:47.352012 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.352672 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:47.365561 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:47.378167 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:47.380714 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:47.435737 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:47.459355 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:47.468485 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.468781 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.473311 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:47.476733 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:47.479386 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:47.484019 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:47.524587 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:47.575449 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:47.580698 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:47.582029 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.591308 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:47.591480 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:47.592803 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.593061 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:47.626222 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:47.635340 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:47.642583 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.643113 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.645805 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:47.659425 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:47.760316 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.760864 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:47.762697 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:47.771382 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:47.771453 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.773341 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:47.783459 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:47.789121 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:47.799165 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.807859 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.808542 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:47.808616 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:47.808883 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 14:02:47.813899 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:47.814853 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:47.824532 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:47.824553 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:02:47.825184 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:47.925794 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:47.927116 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:47.928710 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:47.930333 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:47.932183 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:47.949596 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:47.954977 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:02:48.032423 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:48.081386 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:48.112555 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:48.120893 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:48.121287 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.122828 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:48.124701 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:48.125046 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.125126 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:48.231693 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:48.233891 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.239405 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:48.243978 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:48.244272 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:48.271502 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.285813 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.286003 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:48.291425 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:48.294281 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.301209 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:48.302953 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:48.323462 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:48.324839 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:48.394569 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.397351 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.405644 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:48.405660 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:48.416627 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:48.416716 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:48.460069 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:48.498216 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.499203 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:48.500675 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:48.500993 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:48.502811 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:48.518568 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:48.547734 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:48.597198 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:48.597929 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.600612 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.601914 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:48.607401 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:48.607793 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:48.639156 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:48.656549 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:48.660732 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:48.667629 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.668931 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:48.681918 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:48.683168 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:48.758678 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:48.766376 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:48.771430 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.772184 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.774700 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:48.776480 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:48.780100 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:48.790410 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:48.821447 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:48.824678 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:48.825982 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:48.828001 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:48.831022 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:48.839162 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:48.848225 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:48.944049 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:48.968791 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:49.020272 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:49.053737 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:49.117509 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:49.121074 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.129812 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:49.130895 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.142606 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:49.145826 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:49.201431 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.203711 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:49.204054 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.205008 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:49.213320 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:49.225281 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:49.228286 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:49.268445 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:49.274600 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:49.290501 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:49.291501 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:49.294999 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:02:49.295799 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:49.298116 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:49.307183 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.309799 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:49.310428 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:49.316422 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:49.362532 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:49.364614 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:49.367516 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:49.371784 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.374311 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.375968 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:49.383814 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:49.383963 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:49.441149 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:49.480293 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:49.481550 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:49.487179 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:49.488035 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:49.489318 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.491505 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.515883 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:49.536444 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:02:49.583389 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:49.602903 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:49.603205 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.603291 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:49.626394 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:49.630266 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:49.642486 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:49.656423 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:49.671626 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:49.673676 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:49.674124 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:49.675057 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.675796 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:49.698324 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:49.707832 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:49.761412 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:49.766496 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:49.767942 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.772976 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.781404 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:49.793279 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:49.793587 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:02:49.809839 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:49.822822 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:49.843344 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:49.844238 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:49.846784 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:49.848801 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:49.849799 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:49.856881 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:49.889284 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:49.898787 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:50.081087 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.083367 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:50.084643 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.092106 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:50.100049 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:50.111625 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:50.188234 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:50.194196 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:50.194551 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.198872 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:50.201176 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.220410 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:50.258229 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:50.258889 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:50.260404 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.263207 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:50.272090 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.273551 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:50.280468 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:02:50.339115 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:50.341567 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:50.360376 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:50.361681 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:50.370272 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.375982 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.378877 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:50.395117 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:50.395421 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:02:50.428810 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:50.431972 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:50.439167 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.439643 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:50.442633 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:50.443979 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:50.454823 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:50.459805 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:50.538264 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:50.539770 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:50.541939 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.546103 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:50.551460 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:50.557309 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 163 similar messages]
I20240503 14:02:50.558100 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:50.561581 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:50.626461 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:50.648991 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:50.650892 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:50.657555 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.660791 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:50.660827 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:50.678309 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.682001 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:50.736104 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.741974 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:50.744776 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:50.746970 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.754864 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:50.763376 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:50.799445 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:50.826476 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:50.844982 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:50.848760 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.849886 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:50.853328 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.853510 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:50.866223 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:50.889423 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:50.959355 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:50.983919 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:50.990748 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:50.991019 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:50.999989 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:51.003522 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:51.010082 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:51.048926 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:51.097992 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.100386 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:51.105656 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:51.109424 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.113795 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:51.132126 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:51.193722 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:51.201076 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:51.203450 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:51.216181 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:51.216917 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.218026 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:51.219398 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:51.253420 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:51.267437 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:51.271486 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:51.273701 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:51.274314 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:51.276695 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.296921 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.350391 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:51.352221 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.352733 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.359934 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:51.362366 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:51.367372 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:51.370956 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:51.382221 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:51.433657 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:51.452672 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.454059 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:51.455683 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.457510 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:51.458066 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:51.459393 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:51.464772 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:02:51.472506 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:51.557811 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.558120 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:51.561560 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:51.562333 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.576500 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:51.577564 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:51.593008 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:02:51.663344 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:51.671787 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:51.676157 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:51.676432 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.679356 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.681152 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:51.681501 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:51.684379 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:51.769862 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:02:51.780905 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:51.781898 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:51.782016 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:51.783700 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:51.784803 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.787462 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:51.793351 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.795027 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:51.800654 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:02:51.812566 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:02:51.812709 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:02:51.828022 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:51.878293 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:51.879079 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:51.883728 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.891083 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:51.891734 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:51.899291 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:51.983826 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:51.993183 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:51.994632 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:51.999037 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:52.000310 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.006202 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:52.031708 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:52.110450 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:52.115129 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:52.115631 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.117161 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:52.118491 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:52.121436 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:52.124250 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:52.158560 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 14:02:52.159940 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:52.163015 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:52.182263 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:52.183610 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:52.186017 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.198197 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:52.202863 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:52.207552 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:52.223380 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:02:52.224023 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:02:52.230953 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:52.256572 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.258067 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:52.258423 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.260933 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:52.268800 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:52.284298 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:52.329159 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:52.331948 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.333463 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.333576 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:52.339674 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:52.350005 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:52.377573 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:02:52.384271 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:52.435292 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:52.436223 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.436465 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:52.436658 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:52.439805 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:52.447628 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:52.567665 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:52.574661 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:52.580729 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:52.583065 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:52.583693 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:52.583920 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.600368 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:52.664109 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:52.681741 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:52.684188 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:52.687425 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:52.688378 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.688602 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.691365 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:52.734587 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:52.776316 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.776320 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.776685 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:52.777125 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:52.778882 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:52.782788 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:52.812227 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:02:52.812908 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:52.877089 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:52.877362 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:52.888835 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.889374 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:52.890492 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:52.893898 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:52.911346 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:02:52.946018 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:02:52.947798 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:02:52.949067 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:52.987763 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:52.992244 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:52.996984 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.001638 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.003357 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:53.020498 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:53.059732 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:53.061064 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.066614 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:53.079660 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:53.080183 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:53.081418 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:53.097894 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:53.133976 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:53.161913 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:53.168766 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:53.172262 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.178284 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.178548 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:53.179611 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:53.248508 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:53.252738 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:53.253747 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.256047 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:53.256565 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.283780 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:53.289364 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:02:53.330366 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:53.340176 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:53.340570 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.343345 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.343614 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:53.355381 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:53.357360 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:53.487500 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:02:53.525629 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:53.575634 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:53.576799 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:53.579994 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.585763 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:53.586062 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:53.604287 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.688084 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:53.691696 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.694466 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:53.699295 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.699533 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:53.701583 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:53.707751 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:53.708460 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:53.718266 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:53.738101 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:53.746579 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:53.747803 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.748862 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.751040 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:53.755287 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:53.760769 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:53.824088 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:02:53.831451 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:53.858528 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:53.861770 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.864733 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:53.865705 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:53.872210 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.874619 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:53.912443 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:53.967499 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:53.971431 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.972092 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:53.972878 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:53.992538 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:53.992563 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:54.051713 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:54.076825 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.082496 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:54.087549 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:54.090319 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:54.092110 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.100937 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:54.108407 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:54.124758 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:54.137706 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:54.142268 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:54.142654 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:54.146801 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.154335 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.159976 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:54.161880 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:54.212711 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:02:54.232745 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:02:54.271098 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:54.276924 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:54.277257 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:02:54.278694 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:54.280418 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:54.284663 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.289403 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:54.293154 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:54.293666 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.315106 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:54.321696 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:54.330924 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.332713 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:54.332998 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.349916 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:54.428006 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:02:54.522244 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:02:54.582130 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:54.639639 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:54.645009 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.647799 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:54.661192 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.673588 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:54.674222 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:54.677084 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:54.682562 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:54.688321 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:54.689039 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:54.701380 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:54.707849 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:54.708333 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.712229 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:54.724225 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:02:54.753547 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:02:54.803323 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:54.813642 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:54.816385 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.829546 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:54.843159 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:54.844602 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:54.848093 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:54.856148 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:54.856504 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:54.864097 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:54.866879 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:54.876724 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:54.878088 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:54.884768 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:54.892762 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:54.996131 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:55.001215 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.002517 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:55.004223 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:55.004673 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.008524 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:55.026479 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:55.105476 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:55.106957 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:55.108985 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.109781 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:55.113801 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:55.122030 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.127378 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:55.208395 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:55.210553 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.212172 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:55.215644 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:55.223476 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.233855 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:55.274078 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:55.275497 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:55.277797 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:55.280252 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:55.283752 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.291064 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:55.306216 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.371181 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:55.373853 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.375058 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:55.375355 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:55.375545 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:55.392920 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:55.393978 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:55.417140 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:02:55.574427 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:55.594044 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:55.595633 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.597357 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:55.597628 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:55.598001 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:55.605908 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:55.612515 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:55.626736 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240503 14:02:55.627489 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:02:55.658699 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:55.679575 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:55.716324 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.716423 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:55.719950 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.724138 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:55.732919 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:55.734648 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:55.735532 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:55.749764 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:55.751403 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:55.758605 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:55.761863 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.764691 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:55.767612 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:55.769450 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:55.870862 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:55.884634 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:55.890839 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:55.895462 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.895852 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.898222 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:55.901863 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:55.912778 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:55.919379 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.921895 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:55.926725 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:55.936210 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:55.949390 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:56.052683 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.052935 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:56.056519 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:56.060036 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:56.071828 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:56.080521 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.168365 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:56.168370 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:56.170389 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:56.170979 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.171574 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.174242 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:56.176937 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:56.217371 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:56.256513 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:56.261922 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:56.269706 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:56.272727 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:56.273093 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.275714 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:56.276554 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:56.278667 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:56.286757 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:56.287204 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:56.378731 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.380318 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:56.380920 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:56.381944 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:56.383268 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:56.396766 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:56.401929 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:56.486383 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.493414 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:56.496266 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:56.498461 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:56.506335 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:56.506947 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:56.508088 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:56.534637 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:02:56.575598 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:56.593235 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:56.596045 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:56.596637 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:56.598296 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:56.601404 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:56.607225 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:56.612344 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:56.612844 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:56.641625 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:56.710490 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:56.713322 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.714983 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:56.734141 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:56.741698 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:56.742928 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.745849 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.757601 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.758523 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:56.759375 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:56.764389 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:56.776137 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:02:56.789774 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:56.796944 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:56.813113 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:56.845574 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:56.885824 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:56.887807 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:56.896625 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.897084 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:56.913003 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:56.914417 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:56.936494 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:56.938808 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:56.939205 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:56.958137 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:56.958863 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:56.960317 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:56.974635 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:57.008615 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:02:57.015468 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:57.066921 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.069453 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:57.069629 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.073323 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:57.073570 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:57.076424 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:57.155095 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:02:57.224758 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:02:57.236294 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 14:02:57.245194 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:57.251260 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:57.252125 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:57.253176 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.259799 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:57.263397 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:57.273507 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:57.298520 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:57.315871 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:57.358229 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:57.360378 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:57.362324 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.364699 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.384435 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:57.392915 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:57.444169 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:57.444902 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:57.447474 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.459419 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:57.459600 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:57.459825 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:57.461256 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:57.534559 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:57.549484 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:57.555016 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:57.555483 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:57.556061 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.559064 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:57.574874 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:57.623575 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:57.632560 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:57.668394 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:57.670214 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:57.670506 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.672053 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.678020 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:57.679409 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:57.736476 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:57.737532 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.739632 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:57.749392 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:57.764302 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:57.767581 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.842510 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:57.845180 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:57.847157 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:57.855540 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:57.858211 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:57.861137 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.879990 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:57.895637 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:57.899555 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:57.902036 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:57.913640 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:57.915832 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 14:02:57.925787 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:57.926434 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:57.929302 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:58.018482 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:58.018517 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:58.018951 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:58.022635 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:58.023973 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:02:58.024627 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:58.027917 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:58.029383 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.030229 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:58.111542 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:58.117900 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:58.221617 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:58.221880 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:58.222554 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.224498 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:58.242013 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:58.242971 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:58.250707 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:02:58.307660 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:02:58.315299 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:58.348340 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:58.348680 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.350494 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:58.351663 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:58.358464 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:58.360405 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:58.367883 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:58.377233 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:58.378005 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:02:58.381539 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.395627 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:58.398238 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.420773 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:58.440107 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:58.496701 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:02:58.509464 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:58.512779 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.514104 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:58.514307 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.524021 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:58.527918 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:58.552698 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:58.566885 16946 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:02:58.567006 16946 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:02:58.567523 16946 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:02:58.568023 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:02:58.568152 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:02:58.568228 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:02:58.568280 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:58.569180 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:58.569249 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:02:58.569377 16946 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:02:58.614879 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:58.618711 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:58.618701 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.628924 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:58.631707 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.640004 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:58.726718 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:58.731523 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:58.741916 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:58.744253 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.746814 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.746886 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:58.791066 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:58.803560 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:58.804245 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.806808 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.812351 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:58.822407 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:58.827366 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:58.828011 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:58.835937 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:58.907131 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.911290 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:58.914206 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:58.916441 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:58.921192 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.938669 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:02:58.946739 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:58.970113 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:58.978561 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:58.979202 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:58.984642 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:58.985926 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:58.986347 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:59.011938 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:02:59.014364 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:02:59.088691 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:02:59.142338 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:02:59.175021 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:59.198421 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:02:59.206259 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:02:59.225910 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:02:59.227576 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:02:59.273625 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:59.305235 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:59.306838 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:02:59.307760 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:59.307999 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.323134 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.324538 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:59.326026 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:59.327160 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:59.342244 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:59.348464 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:59.361740 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.387995 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:59.388732 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:59.422246 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:02:59.496866 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.499737 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:59.502264 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:59.502799 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.508540 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:59.520287 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.521817 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:59.536633 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.539916 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:02:59.564314 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:02:59.572129 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:59.575013 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:59.576732 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:02:59.646381 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:02:59.664309 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:59.668094 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.671989 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:59.672636 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:59.674479 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:59.691803 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:02:59.697664 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:59.713044 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:02:59.734678 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:02:59.757458 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:02:59.779431 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.779649 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:59.784669 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.791118 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:59.792869 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:59.799284 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:02:59.867565 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:02:59.891278 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:59.892006 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.893045 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:59.895440 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:59.902108 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:02:59.909469 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:02:59.950383 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:02:59.962714 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:02:59.963752 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.964326 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:02:59.966624 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:02:59.978333 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:02:59.985563 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:00.056396 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:00.056540 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:00.057791 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.058758 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:00.064823 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.071872 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:00.170619 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:00.175015 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:00.175104 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:03:00.195966 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:03:00.209535 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:00.262565 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:03:00.286937 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:00.289559 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:00.290805 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:00.295547 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.296418 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.315519 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:00.397840 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.400259 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.402585 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:00.415182 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:00.422991 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:00.423497 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:00.425442 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:03:00.475713 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:00.478458 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:00.482928 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.483179 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:00.483178 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:00.491437 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.577275 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.580703 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:00.582873 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:00.592852 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:00.597563 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:00.599743 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:00.632108 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:00.634246 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 158 similar messages]
I20240503 14:03:00.645107 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:00.645306 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:00.645752 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:00.648814 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.653208 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:00.679553 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:00.738556 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:00.757081 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:03:00.770339 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:00.775454 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:00.777515 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:00.777909 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:00.781293 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.783766 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.789409 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:00.814249 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:03:00.836393 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:03:00.897632 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:00.900238 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.905694 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:00.909353 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:00.912122 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:00.913332 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:00.921124 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:01.009711 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:01.016090 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:01.018015 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:01.018474 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:01.023726 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.030086 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:01.121910 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:01.125897 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:01.127990 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:01.129531 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.130286 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.131608 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:01.131978 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:01.171365 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:01.173014 19556 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:03:01.173149 19556 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:03:01.173626 19556 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:03:01.174590 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:01.174932 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:01.175004 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:03:01.175125 19556 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:03:01.198341 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:01.204982 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:03:01.232266 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:01.234510 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:01.237787 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.238590 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:01.243445 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:01.261207 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.347604 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.350648 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:01.351413 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:01.352283 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.357255 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:01.362217 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:01.369426 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:01.391853 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:01.476641 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:01.477937 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.479826 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.480612 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:01.481415 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:01.489832 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:01.525532 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:03:01.534749 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:01.536618 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:01.545135 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.545661 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.545697 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:01.554705 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:01.577221 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:01.639395 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:01.659600 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:01.662144 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:03:01.666239 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:01.668200 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:01.670986 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.671694 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:01.672048 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:01.696831 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.701608 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:01.705643 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:01.707788 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:01.710778 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:01.721484 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:03:01.721602 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:01.723529 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.746003 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.749207 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:01.766352 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:03:01.792388 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:01.840032 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:01.841912 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.842859 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:01.848800 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:01.856017 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:01.856410 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:01.865124 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:01.934530 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:03:01.997756 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:02.040182 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:02.041921 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.042191 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.044580 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:02.046976 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:02.051192 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:02.142473 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:02.143580 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:02.161926 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:02.167510 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.167201 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:02.171129 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:02.176906 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.207245 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:02.215039 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:02.221824 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:02.229543 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.232082 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:02.232363 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:02.238065 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:02.243526 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:02.284265 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:03:02.285508 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:02.317348 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:02.319514 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:02.340003 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:02.343511 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:02.346414 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:02.347172 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.351215 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.355190 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:02.456028 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.457230 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:02.458518 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.469523 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:02.472477 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:02.480722 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:02.517354 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:02.556074 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:02.572219 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:02.583226 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:02.592263 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:02.594112 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:02.595136 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.601478 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.616982 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:02.627149 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.634658 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:02.637147 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:02.639075 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:02.659114 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:02.667454 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:02.679574 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:02.704932 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:02.740171 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:02.779089 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:02.779974 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.780122 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.781510 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:02.790416 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:02.790774 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:02.804116 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.811842 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:02.812336 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:02.815403 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:02.829435 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:02.839179 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:02.901602 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:02.914829 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:03.079618 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 14:03:03.153055 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:03.153801 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.159838 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:03.160907 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:03.163358 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.190286 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:03.199420 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:03.203106 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:03.206529 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.210621 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:03.219702 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:03.229830 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:03.249709 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:03.250002 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:03.290444 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:03.317130 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:03.317843 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:03:03.319010 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:03:03.331725 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:03.332789 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:03.335608 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:03.335718 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.353268 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:03.354959 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.357949 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:03.367615 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:03.383095 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.387392 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:03.391765 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:03.396031 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.425191 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:03.532377 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:03.535467 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.539023 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:03.540794 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:03:03.541725 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:03.541730 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:03.551445 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:03.647358 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:03.650359 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:03.650677 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:03.653548 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.655692 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:03.656422 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.657049 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:03.696851 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:03:03.710691 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:03.757810 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:03.760676 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:03.766294 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.767537 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.767619 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:03.792587 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:03.805394 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:03.820072 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:03.822319 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:03.825011 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:03.831439 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:03.835140 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:03.835438 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:03.835675 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:03.855496 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:03.938410 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:03.938426 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:03.939942 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:03.940002 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:03.940789 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:03.954942 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:04.030495 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:04.031275 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:04.098644 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:03:04.120369 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:04.130431 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:04.134101 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:04.147130 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.147840 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:04.150342 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:04.151400 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:04.153913 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.163748 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:04.183260 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:04.184811 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:04.280064 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:04.283282 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:04.288173 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:04.291530 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:04.295471 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:03:04.295996 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:04.296782 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:04.297519 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:04.298720 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.300083 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.302912 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:04.313232 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:04.318090 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:04.333931 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:04.340996 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:04.435715 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:04.467607 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:04.468473 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:04.476266 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:04.481280 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:04.482961 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:04.484323 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:04.485473 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:04.490128 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.493176 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:04.511984 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.512024 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.513947 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:04.535565 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:04.583732 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:04.628079 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.629156 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.633580 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:04.634402 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:04.635025 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:04.643148 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:04.658385 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:04.721179 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:03:04.733916 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:03:04.735623 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:04.737690 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:04.749020 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:04.750607 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:04.751705 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.751809 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:04.752702 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.760813 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:04.787022 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:04.849098 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:04.854339 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:04.859408 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.868073 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:04.869700 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:04.870134 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:04.937006 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:04.969702 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:04.974284 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.976740 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:04.980140 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:04.983462 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:04.997169 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:05.057683 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:05.098075 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:05.104350 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:05.105211 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.106628 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.107254 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:05.123415 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:05.123404 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:05.141511 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:05.187744 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:05.214474 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:05.217314 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.219806 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:05.223632 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:05.229157 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:05.231674 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:05.315482 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:05.322345 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:05.326131 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:05.327745 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:05.335903 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:05.348616 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.354192 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.395195 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:05.403075 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:05.404610 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:05.413040 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:05.420518 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.443138 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:05.454929 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:05.489197 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:05.493551 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:05.494565 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:05.496280 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:05.500672 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:05.510644 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:05.511483 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:05.557559 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:05.576364 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:05.580974 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.581813 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:05.582559 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:05.594281 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.601904 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:05.623385 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:03:05.665117 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 161 similar messages]
I20240503 14:03:05.693583 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:05.698298 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.701367 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:05.705418 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:05.712179 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.712865 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:05.724709 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:05.729964 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:05.811586 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:05.811691 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.812763 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.814003 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:05.820079 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:05.833906 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:05.898214 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:05.927225 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:05.929627 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.929961 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:05.930409 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:05.938905 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:05.951403 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:05.960264 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:03:06.006063 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:06.039453 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.041123 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:06.041388 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.045320 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:06.054352 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:06.059386 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:06.077350 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:03:06.161259 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.161448 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:06.163977 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:06.165074 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:06.183604 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:06.183733 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.185348 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:06.273818 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:06.276384 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.281092 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:06.283418 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:06.301626 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:06.313858 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:06.322427 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.329095 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.333812 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:06.336864 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:06.343693 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.349110 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:06.365692 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:06.365887 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:03:06.437645 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:03:06.448583 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:06.458664 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:06.458973 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:06.460044 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:06.467438 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.476745 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:06.488243 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.507038 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:06.511572 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:06.520220 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:06.520969 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:06.522840 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:06.527662 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:06.537425 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.544905 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:06.587309 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:06.626521 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:06.629217 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:06.632246 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.635860 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:06.639310 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.641361 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:06.648222 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:06.656380 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:03:06.679646 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:03:06.747941 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:03:06.751303 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:03:06.805764 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:06.813284 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.818516 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:06.818567 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:06.819254 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.832891 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:06.847632 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:03:06.928231 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:06.929617 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:06.932163 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:06.932204 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.937264 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:06.945333 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:06.946494 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:06.989892 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:07.010490 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:07.011797 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:07.013953 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.014397 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:07.014843 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:07.019862 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:07.109706 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:07.121104 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:07.122862 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.124956 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:07.126374 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:07.136085 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.143435 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:07.151062 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:03:07.162135 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:07.229313 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:07.246266 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:07.250367 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.250551 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:07.251283 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.254451 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:07.256920 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:07.316673 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:07.318202 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:07.319625 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.319696 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.320595 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:07.328752 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:03:07.330099 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:07.337391 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:07.377749 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:03:07.416496 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:07.422472 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:07.423532 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:07.424676 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:07.426458 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.427479 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.432976 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:07.449556 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:07.490003 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:07.497208 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:07.498342 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.499384 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.517803 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:07.519467 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:07.525415 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:07.535638 19556 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:03:07.535768 19556 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:03:07.536149 19556 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:03:07.537107 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:07.537233 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:07.537302 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:03:07.537475 19556 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:03:07.568172 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:07.582154 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.582531 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:07.582981 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.589084 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:07.594753 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:07.609745 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:07.616591 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:07.701457 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:07.735078 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:07.808437 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.809036 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:07.812145 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:07.812422 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:07.820624 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.835363 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:07.862246 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:07.938213 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:07.942597 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:07.947127 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.948518 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:07.953301 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.953903 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:07.968781 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.975092 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:07.975517 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:07.978503 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:07.992615 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:07.994000 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:08.000785 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:08.038095 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:08.050853 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:08.107836 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.107919 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:08.108268 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.113406 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:08.116750 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:03:08.125939 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:03:08.128111 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:08.128489 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:08.219877 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:08.220861 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:08.226608 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.228216 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:08.232143 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.233471 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:08.254120 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:03:08.276758 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:08.338104 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:08.345355 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:08.358155 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:08.358819 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.361987 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:08.363704 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:08.373198 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 33 similar messages]
I20240503 14:03:08.377780 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:08.385118 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:08.400923 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.405963 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:08.412153 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.413595 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:08.455446 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:08.531196 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.531333 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:08.543114 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:08.543491 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:08.545120 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:08.557193 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.557484 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:08.558429 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.572705 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:08.574023 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:08.577035 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:08.582785 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:08.592080 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:08.594934 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:08.741542 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:08.809847 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:03:08.848289 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:03:08.885322 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:08.901832 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:08.907685 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:08.910279 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.911307 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.913121 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:08.918717 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:08.934222 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:08.963594 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:08.966851 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.967088 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:08.974944 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:08.981616 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:08.982941 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:08.987483 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:09.026239 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:09.034745 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:03:09.060842 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:03:09.072366 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:03:09.080932 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:09.099197 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:09.099499 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:09.111117 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:09.111135 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:09.112275 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:09.113056 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:09.126996 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:09.129261 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:09.134440 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:09.134804 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:09.141191 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:09.152827 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:09.176389 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:09.180011 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:09.263799 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:09.264686 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:09.265741 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:09.267607 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:09.268404 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:09.281062 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:09.303653 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:03:09.318411 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:09.370443 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:09.372424 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:09.375353 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:09.376503 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:09.380965 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:09.390130 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:09.396571 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:09.400032 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:09.468564 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:09.499435 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:09.508677 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:09.511335 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:09.519508 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:09.519763 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:09.520205 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:09.522598 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:09.537552 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:09.540946 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:09.555363 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:09.574323 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:09.578930 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:09.656224 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:09.663374 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:09.664971 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:09.671785 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:09.672600 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:09.673298 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:09.685940 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:09.755631 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:03:09.773352 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:03:09.822364 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:09.864354 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:09.865604 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:09.866798 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:09.868155 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:09.872018 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:09.873880 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:09.882217 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:03:09.887298 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:09.893071 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:09.966827 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:03:09.975984 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:09.994028 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:09.998605 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.008229 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:10.011960 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.012874 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:10.030511 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:10.058336 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:10.062455 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:10.067719 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.071455 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:10.072901 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.073583 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:10.156870 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:10.158773 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:10.161731 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.166603 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.169636 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:10.172050 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:10.183876 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:10.187201 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:10.230741 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.230965 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:10.235746 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:10.239534 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:10.255048 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.256692 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:10.317700 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.318758 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:10.319945 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.323369 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:10.330520 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:10.330787 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:10.350009 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:10.360638 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:10.377193 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:10.439848 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:10.440634 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:10.443980 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:10.451395 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:10.452174 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:10.452711 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.453145 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:10.499400 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:10.542459 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:10.548754 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:10.553314 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:10.556015 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:10.559901 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.560665 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:10.562541 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:10.641155 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:10.666111 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:10.667730 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.669044 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.670096 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:10.670567 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:10.671119 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:10.697772 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 159 similar messages]
W20240503 14:03:10.759732 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:10.772972 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:10.774245 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.776635 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:10.778304 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:10.778821 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:10.787643 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.885003 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:10.886365 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:10.889125 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:10.889940 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:10.893569 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:10.895429 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:10.923728 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:10.930020 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:10.964569 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:03:11.006448 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:11.026878 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:11.034555 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:11.034590 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:11.037366 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:11.040799 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:11.050155 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.052137 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:11.053475 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:11.053731 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:11.054741 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:11.070897 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.072041 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.095631 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.202414 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:11.206657 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:11.209306 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:11.214916 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:11.216585 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.218550 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:11.232911 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:03:11.238085 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.243803 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.244338 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:11.249526 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:11.260763 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:11.270231 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:11.312880 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:11.357578 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:11.365262 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:11.368569 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.370355 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:11.371567 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.375571 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:11.446444 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:03:11.520911 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:03:11.532862 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:11.554822 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:11.570139 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:11.572914 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:11.575528 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:11.578240 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:11.578517 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.581327 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:11.583123 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:11.588475 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:11.703480 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:11.705387 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.707129 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:11.715299 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:11.719732 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.721418 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:11.731621 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:11.731976 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:11.732874 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:11.741062 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:11.741143 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:11.741139 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:11.743718 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:11.754987 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:11.755226 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:11.768203 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:11.848191 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:11.861613 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:11.862411 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:11.863973 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:03:11.865876 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:11.871917 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.872555 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.881935 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:11.978843 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:11.979508 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:11.982059 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:11.982529 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:11.982990 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:11.985055 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:11.985123 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:11.999344 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:12.111894 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:12.118301 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:12.119047 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:12.124886 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:12.132946 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.133713 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:12.134662 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.136209 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.151552 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:12.155567 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:12.157054 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.157513 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:12.182979 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:12.264357 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:12.279804 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:12.284410 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:12.300410 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:12.305644 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.306020 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:12.312975 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:12.313437 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.315438 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.316287 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:12.319667 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:12.336051 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:03:12.336051 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:12.337145 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:12.337291 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:12.339056 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:12.355062 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:12.356791 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:12.428562 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:12.473989 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:12.502626 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:12.527694 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:12.619340 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:12.682070 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.683696 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:12.684794 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:12.688020 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:12.698007 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:12.701429 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:12.707386 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:12.722582 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:12.723181 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:12.738272 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.739122 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:12.744235 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:12.748356 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.756187 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.852522 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:12.854847 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:12.862442 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:12.865244 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.866621 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.893471 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:12.894230 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:12.916882 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:12.918753 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:12.923789 19982 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:03:12.923919 19982 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:03:12.924445 19982 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:03:12.925472 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:03:12.925590 12036 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:03:12.925659 12036 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:03:12.926831 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:12.926956 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:12.927022 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:03:12.927313 19982 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:03:12.930553 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:12.932843 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:12.933401 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:12.952327 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:13.018138 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:13.024294 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:13.027261 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:13.032716 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:13.033609 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.033624 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.062644 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:13.112339 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:13.120297 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:13.130650 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 14:03:13.146147 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:13.146517 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:13.148754 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.149015 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:13.149339 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:13.149513 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:13.162118 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:13.242417 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:13.275125 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:13.277456 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:13.278697 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.279150 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:13.279174 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:13.282758 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:13.283494 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.286571 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:13.293665 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:13.296708 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.316411 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:13.322031 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.339290 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:13.415182 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:13.431423 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:13.435046 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.436447 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:13.436686 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:13.440975 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.441553 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:13.464874 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:03:13.585944 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:13.593130 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:13.634289 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:13.634951 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:13.641109 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.642855 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:13.662254 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:13.665668 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:13.683780 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:13.756103 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:13.756615 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:13.764722 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:13.771754 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.774890 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:13.786718 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:13.788583 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.788786 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:13.791678 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:13.818704 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:13.829190 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:13.830552 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:13.831197 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.831614 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:13.881935 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:13.901180 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:03:13.920727 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:13.933391 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.934796 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.934937 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:13.953838 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:13.960179 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:13.961017 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:13.961997 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:13.966681 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:13.972101 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:13.997380 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.010429 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:14.015946 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:14.023082 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:14.032790 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:03:14.110580 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.111830 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:14.112313 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.117300 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:14.125461 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:14.130729 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:14.132241 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:14.162873 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:14.204289 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:14.213351 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:14.226656 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:14.227746 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.228233 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:14.230876 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:14.241413 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.241674 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:14.252068 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:14.324730 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:14.329104 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:14.329680 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.331110 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.350858 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:14.351564 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:14.439299 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:14.442165 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:14.442417 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:14.448815 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:14.448832 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:14.449086 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:14.449764 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:14.466181 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:14.511603 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:14.549304 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:14.550266 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:14.553871 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:14.556223 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.558176 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.558864 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:14.585505 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:14.616983 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:14.626740 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:14.659165 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:14.662315 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.663163 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:14.663903 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.668606 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:14.678620 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:14.722560 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:14.739578 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:14.763334 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:14.770371 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:14.776615 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:14.776914 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:14.781634 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.782052 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:14.787309 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:14.791162 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:14.815536 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:14.827419 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.829375 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.831437 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:14.835533 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:14.840654 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:14.851756 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.944746 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:14.944860 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:14.947289 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:14.954282 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:14.956478 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:14.957123 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:14.983728 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:14.988401 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:15.015177 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:15.019399 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.020587 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:15.021389 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.038285 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:15.042016 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:15.068483 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:15.144107 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:15.145562 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:15.146230 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:15.146801 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.147560 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.163120 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:15.165235 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:15.181334 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:15.287755 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:15.294667 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:15.297986 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:15.302649 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.303264 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.315394 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:15.369560 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:03:15.373953 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:15.380700 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:15.390249 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:15.414161 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:15.415952 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.416127 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:15.419001 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:15.432428 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:15.440328 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.495478 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:15.499794 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:15.503527 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:15.504711 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:15.507351 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.511402 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:15.546492 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:15.551124 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:15.567338 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:15.611578 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:15.612608 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.613510 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:15.614131 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:15.615628 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.634887 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:15.734537 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:15.735600 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.739122 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:15.743006 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:15.759335 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.761401 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:15.783077 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:15.786039 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.788654 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:15.791882 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:15.794091 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:15.816026 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:15.822692 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:15.835031 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240503 14:03:15.857193 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:15.919739 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:15.924433 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:15.935288 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.936225 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:15.937319 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:15.938762 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:15.948844 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:15.953490 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:15.953961 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:15.954210 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:15.955631 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:15.960340 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:16.001719 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:16.004160 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:16.086939 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:16.088805 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:16.090317 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:16.093369 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.099886 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.107976 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:16.119944 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:16.124523 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:16.291632 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:16.293762 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.294603 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:16.294615 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:16.296623 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:16.305730 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:16.315690 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:16.316152 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:16.322090 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:16.388120 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:16.421805 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:16.423413 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:16.429898 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:16.438617 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.439674 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:16.440261 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.441326 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:16.444860 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:16.452705 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.454217 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.467388 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:16.507959 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:16.513810 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:03:16.525074 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:16.590461 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.591539 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:16.594058 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.603430 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:16.605158 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:16.605686 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:16.640137 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:16.659377 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:16.678781 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:16.702586 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:16.706184 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:16.707165 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:16.709286 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.713241 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:16.725908 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.832319 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:16.833647 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:16.840874 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:16.842059 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:16.853039 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.854429 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:16.864563 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:03:16.870363 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:16.871726 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:16.872510 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:16.885998 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:16.888840 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:16.907583 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:16.907864 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:16.910038 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:03:16.947417 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:16.962873 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:03:17.000142 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:03:17.001288 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:17.017477 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:17.021036 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:17.033347 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:17.037822 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:17.041803 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:17.050482 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:17.050892 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:17.053511 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:17.060338 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:17.067390 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.069798 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.088964 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.095393 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:17.245522 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:03:17.287853 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:17.327625 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:17.365552 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:03:17.375176 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:03:17.376950 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:17.401183 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:17.404337 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:17.410815 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:17.418234 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.423887 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:17.431862 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.433017 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:17.437424 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:17.461503 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:17.463747 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:17.468053 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.468796 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:17.545480 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:17.579985 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:17.580452 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:17.580485 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:17.581259 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.583191 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:17.590492 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.594054 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:17.600706 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:17.602519 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:17.602941 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:17.611698 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:17.620694 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:17.626540 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:17.633343 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:03:17.638399 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:17.654788 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:17.671434 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:17.734287 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:17.742410 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:17.742800 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:17.743844 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:17.744491 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.751807 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.757408 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:17.757838 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:17.805092 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:17.853652 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.855350 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:17.863667 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.864470 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:17.867383 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:17.882946 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:17.960384 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:17.960855 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:17.962893 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:17.965906 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:17.972498 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:17.979986 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.036046 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:18.036960 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:18.037645 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.044893 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.045248 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:18.052500 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:18.108434 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:18.124449 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.125375 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:18.126374 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:18.135989 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.139391 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:18.140307 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:18.144721 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:03:18.155076 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:03:18.175210 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:18.221879 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:18.280586 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:03:18.293263 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:18.354645 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:18.358297 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:18.359182 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:18.362421 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.363827 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:18.363912 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.481666 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:18.489773 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.490404 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.490787 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:18.491478 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:18.502835 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:18.521240 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:18.521476 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:18.522861 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:18.524304 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:18.532229 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:18.533742 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:18.544728 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:03:18.563875 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:18.630770 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:18.659895 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:18.663118 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:18.664672 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:18.665163 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.673578 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:18.676789 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.706827 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:18.711490 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:18.718381 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.719748 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:18.720314 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:18.721298 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.725759 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:18.733546 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:18.744052 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:18.811129 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:18.814141 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:18.824559 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.826220 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:18.827661 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:18.828563 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:18.828867 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.830794 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:18.868613 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:18.941663 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:18.942466 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:18.945746 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.947515 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:18.963833 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:18.965978 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:19.053499 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:19.054873 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:19.059036 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.059607 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:19.059998 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:19.073716 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:19.078629 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:19.096467 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:03:19.130180 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:19.161968 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.162240 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:19.164786 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.171327 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:19.184293 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:19.190361 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:19.196413 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:03:19.227941 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:19.280332 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.281124 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:19.284418 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:19.286497 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:19.291551 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:19.293023 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.391626 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:19.393301 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:19.395185 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:19.403802 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:19.407755 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:19.409957 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:19.415195 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:19.416303 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:03:19.417953 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:19.422253 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:19.466171 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:19.507385 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:19.509284 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:19.518369 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:19.519165 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:19.520701 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.521780 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.549391 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:19.569144 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:19.582073 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.582307 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:19.584728 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:19.586244 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:19.588027 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:19.590173 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:19.673956 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:19.684376 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:19.686652 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:19.694051 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.697407 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:19.697654 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:19.711400 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:19.744294 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:19.760810 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:19.761207 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.762344 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:19.765026 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.771384 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:19.771693 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:19.782514 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:03:19.821908 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:19.833731 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:03:19.851964 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:19.878540 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.878959 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:19.879681 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:19.880892 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:19.881825 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:19.883211 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:19.893944 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:19.950621 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:20.017553 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:20.018093 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:20.019094 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.020011 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.021428 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:20.035459 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:20.050400 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:20.120510 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:03:20.126322 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:20.142163 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.142980 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:20.144572 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:20.144943 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:20.145038 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:20.174968 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:20.177304 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.234514 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.236663 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:20.237558 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:20.237625 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.238811 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:20.253470 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:20.274377 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:03:20.320600 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:20.346786 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:20.352247 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.353509 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:20.356299 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:20.363924 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:20.367792 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:20.446978 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:20.452183 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:20.481818 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:20.483665 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.497859 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:20.498003 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:20.498804 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:20.506281 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.507604 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:20.508467 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:20.516033 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:20.527909 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:20.528720 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:20.552619 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:20.563942 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:20.600232 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:03:20.628252 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:20.653451 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:20.657814 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:20.658327 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.658686 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:20.667994 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.668854 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:20.720547 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:20.723287 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.725924 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:20.734800 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:20.739284 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.746817 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:20.800503 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:03:20.808140 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:20.837899 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 164 similar messages]
I20240503 14:03:20.838883 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.840117 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:20.840678 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:20.858117 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:20.863183 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:20.871223 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:20.916843 19982 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:03:20.917021 19982 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:03:20.917572 19982 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:03:20.918289 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:03:20.918422 12036 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:03:20.918494 12036 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:03:20.918604 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:20.918921 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:20.918983 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:03:20.919123 19982 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:03:20.919394 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:20.992643 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:21.054438 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.055786 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:21.058593 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:21.059413 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:21.076524 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:21.077068 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:21.136606 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:21.183837 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:21.186668 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.201392 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:21.201803 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.202214 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:21.212780 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:21.227340 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:03:21.234548 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:21.236071 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:21.237430 26869 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:03:21.237556 26869 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:03:21.238013 26869 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:03:21.240713 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:21.241250 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:21.241320 12103 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:03:21.241654 26869 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:03:21.245355 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:21.246287 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.247630 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:21.249426 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.266494 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:21.268951 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:03:21.326117 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:21.356046 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.356367 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:21.357777 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:21.357885 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:21.358165 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.363400 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:21.387066 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:21.474546 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:21.474786 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.475209 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:21.476040 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.482416 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:21.501685 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:21.532312 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:21.569344 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:21.597038 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:21.599619 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.601608 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.605021 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:21.618463 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:03:21.619195 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:21.633335 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:21.636215 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:21.650092 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.655292 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:21.659204 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.663750 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:21.674897 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:21.729415 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:21.767453 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:21.773102 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:21.773288 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:21.779395 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.782063 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:21.791725 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:03:21.806191 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:21.809232 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:21.809626 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:21.811370 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:21.815073 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:21.826189 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:21.839569 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.840210 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:21.856897 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:21.858855 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:21.999176 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:03:22.025828 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:22.037720 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:03:22.044492 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:03:22.080513 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:22.153589 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:22.153941 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:22.159607 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:22.160197 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.161959 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.170918 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:22.205117 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:22.210126 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:22.218964 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.220479 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:22.227197 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:22.239375 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:22.308128 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:22.346557 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:22.350983 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.352943 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:22.356704 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.357087 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:22.366536 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:22.368573 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:22.369705 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:22.370616 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:22.371994 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:22.380823 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:22.383826 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:22.389469 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 14:03:22.390199 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:22.398931 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:22.431277 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:22.439322 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:03:22.453410 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:22.512190 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:22.520835 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.522307 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:22.523301 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:22.524426 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:22.525198 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.529402 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:22.545783 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:03:22.558830 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:03:22.562853 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:22.630374 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:22.638638 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.640494 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:22.641593 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.641688 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:22.655177 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:22.727396 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:22.750645 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:22.751672 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.754951 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.763902 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:22.769704 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:22.770401 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:22.817211 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:22.823132 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.825170 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:22.827739 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:22.830468 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.840009 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:22.842797 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:22.925662 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.925928 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:22.927138 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:22.928351 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:22.931413 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:22.931416 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:22.962844 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:22.975381 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:23.098562 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:03:23.128962 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:23.147342 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.147804 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:23.150758 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:23.151635 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:23.159385 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:23.168952 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:23.170863 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:03:23.216732 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:23.256486 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:23.258229 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:23.262802 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.266463 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.272367 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:23.277271 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:23.326284 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:23.328960 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.329447 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:23.331140 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:23.335122 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:23.341220 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:23.353454 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:23.368755 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:03:23.447789 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:23.459841 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:23.462627 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:23.475812 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:23.477022 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:23.478111 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:23.478840 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:23.479789 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.487682 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:23.494129 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.496438 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.509277 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:23.519995 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:23.581660 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:23.621467 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:23.621932 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:23.629109 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:23.630390 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.630801 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:23.650569 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:23.686477 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:23.690613 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 14:03:23.740974 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:23.741746 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:23.744963 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:23.746197 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.746987 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:23.749542 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.763767 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:23.842901 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.844085 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.844759 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:23.859822 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:23.862244 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:23.869958 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:23.909629 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:23.961290 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:23.961604 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:23.968868 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:23.978672 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:23.979279 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:23.979305 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:23.982472 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:24.017190 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:03:24.021860 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:24.072786 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:24.078748 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.079465 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:24.080358 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:24.081405 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.093025 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:24.120808 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:24.166278 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:24.183400 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.183708 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:24.187081 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:24.194072 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:24.194836 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:24.195969 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:03:24.202767 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:24.236569 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:03:24.258888 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:03:24.279732 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:03:24.291249 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:24.299393 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:24.299680 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:24.306128 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.311851 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.324537 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:24.327145 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:24.372040 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:24.372874 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:24.375653 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:24.378852 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:24.379360 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.379601 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:24.389134 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:24.428488 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:24.497522 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:24.502003 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:24.509321 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:24.512861 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:24.514060 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:24.515619 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.523316 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.538166 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:24.546754 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:24.549444 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.553023 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:24.554610 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:24.568516 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.588302 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:24.591338 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:24.632838 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:24.676403 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:24.676589 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:24.679646 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.680821 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:24.680869 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.681175 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:24.720129 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:03:24.751988 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:24.787995 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:24.851253 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:24.852293 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.853020 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.864341 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:24.869438 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:24.877956 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:24.883826 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:24.884315 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:24.921036 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:24.941319 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:24.980204 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:24.980298 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:24.985730 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:24.996043 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.004089 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:25.006040 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:25.006119 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:25.061612 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:25.062041 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:25.062228 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:25.063732 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.066988 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:25.084759 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:25.090678 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:25.149071 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:25.200040 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:25.203361 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:25.204035 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.204151 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:25.212329 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:03:25.213119 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:25.213794 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.226449 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:25.321089 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:25.332993 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.336350 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:25.337715 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:25.343079 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:25.348392 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:25.355399 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:25.366201 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:25.375998 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:25.377331 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:25.379678 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.393241 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.403575 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:25.415267 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:25.513928 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:25.520364 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.532238 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:25.532356 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:25.538141 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:25.539868 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:25.542351 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:25.560536 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:25.561568 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.562948 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:25.572978 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:25.582036 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:25.591401 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.685314 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:25.686476 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:25.687251 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:25.691087 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.692322 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:25.705236 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:25.705453 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:03:25.729665 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:25.797981 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:03:25.806851 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:25.808557 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:03:25.860775 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:25.891031 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:25.892371 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:25.896600 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:25.903357 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:25.903910 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:25.906714 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:25.918869 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 160 similar messages]
I20240503 14:03:26.009792 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:26.014034 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:26.025707 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.040627 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.051450 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:26.052289 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:26.058105 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:26.068253 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.069707 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:26.082057 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:26.082671 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:26.092237 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:26.095666 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.184398 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:26.188370 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.192195 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.199554 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:26.207376 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:26.209153 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:26.282879 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:26.297740 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:26.299441 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:26.301496 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:26.302418 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.309697 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:26.313035 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:26.315665 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:26.332579 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.334980 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:26.401365 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:26.408062 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.420485 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:26.422003 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.424911 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:26.434350 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:26.487222 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.488581 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:26.489818 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:26.490411 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.491166 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:26.504837 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:26.507787 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:26.562812 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:03:26.579706 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:26.580040 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:26.582542 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.588541 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.614848 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:26.615660 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:26.637526 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:26.642714 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:26.647289 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:26.647359 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:26.650151 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:26.650355 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:26.656845 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:26.658572 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.672585 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:26.686527 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:26.762598 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:26.865816 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:26.887720 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:26.921744 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:26.925843 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:26.933526 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:26.935449 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:03:26.938972 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:26.941874 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:26.943712 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:26.954222 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:26.957440 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:27.003414 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:27.009135 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.014132 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:27.016825 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:27.017036 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:27.026453 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:27.069686 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:27.097743 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:27.100687 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:27.101151 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.101662 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:27.101771 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.116525 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:27.168939 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:27.169026 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:27.171646 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.173583 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.178596 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:27.193365 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:27.243381 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:03:27.266431 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:27.286967 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:27.287006 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:27.290999 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:27.291822 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:27.297286 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.303988 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:27.304256 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:03:27.306623 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:27.343132 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:03:27.354498 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:27.394496 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:27.407410 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:27.415800 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:03:27.416482 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:27.416872 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:27.417762 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.419798 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.420207 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:27.422106 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:27.473136 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:27.473421 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.479847 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:27.480607 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:27.482525 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:27.483289 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:27.493487 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:27.572631 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:27.573911 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:27.582772 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.587412 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.590291 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:27.596139 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:27.622619 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:27.629415 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:27.633960 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:27.635293 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:27.635632 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:27.635926 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.640689 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:27.654165 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:27.714262 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:03:27.725659 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:27.867795 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:27.893374 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.894529 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:27.897840 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.908658 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:27.909806 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:27.918969 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:27.928021 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:27.979275 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.980670 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:27.981568 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:27.987684 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:27.992833 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:27.994851 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:28.019790 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:28.072263 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.076547 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:28.077100 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:28.078738 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:28.080513 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.086004 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:28.150413 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:28.159830 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.159838 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:28.160826 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:28.167414 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.168107 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:28.188757 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:28.231640 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:28.231784 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.232199 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:28.232702 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:28.241858 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:28.247264 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:28.267316 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:28.273375 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:28.346258 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:28.346319 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.350987 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:28.351506 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:28.351668 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.364467 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:28.364584 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:03:28.400097 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:03:28.440328 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:28.463436 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.469153 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:28.469661 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:28.478775 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:28.492056 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.504103 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:28.516942 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:28.517589 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.517616 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:28.518883 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:28.525506 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.527916 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:28.589823 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:03:28.618295 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:28.624966 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:28.627728 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:28.629405 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:28.630524 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:28.631142 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.639734 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.643136 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:28.693477 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:28.795176 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:28.812656 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.813048 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:28.817265 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:28.818588 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.822033 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:28.827908 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:03:28.832751 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:28.916906 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:03:28.928054 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:28.937885 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:28.941284 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:28.946529 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.949203 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.950261 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:28.968228 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:28.970559 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:28.981027 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:28.983345 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:28.983440 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.009209 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:29.042889 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:29.118790 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:29.121846 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:29.123502 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:29.125005 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:29.126080 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.126138 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:29.136540 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:03:29.139958 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:29.142140 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:29.145718 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.146915 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:29.152269 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:29.153827 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:29.159166 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:29.176113 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:29.187085 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:29.193651 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:29.269598 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:29.274232 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:29.274531 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:29.276037 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.276679 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:29.302235 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:29.361522 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:29.387037 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:29.388681 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.395364 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.399149 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:29.409176 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:29.418574 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:29.444041 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:03:29.450599 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:29.468686 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:03:29.480947 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:29.492211 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.494014 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:29.496140 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.497457 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:29.498381 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:29.502844 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:29.513413 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:29.548075 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:29.592664 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:29.593004 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:29.597563 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.599858 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:29.600144 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:29.607860 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:29.677807 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:29.679937 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:29.703964 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.705617 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:29.705791 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:29.706506 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:29.711084 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:29.716382 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:29.726110 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:03:29.764647 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:29.803750 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:29.808745 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.810680 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:29.811173 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:29.813689 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:29.819944 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:29.839118 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:29.888993 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:03:29.900190 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:29.924288 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:29.928476 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.928874 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:29.929740 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.951269 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:29.958336 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:29.971212 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:29.979190 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:29.986128 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:29.987381 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:29.992784 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:29.996460 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:29.998948 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:30.006604 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:30.049481 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:30.090333 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.090611 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:30.093259 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:30.097572 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:30.099622 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:30.117238 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.141921 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:30.144207 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:30.144891 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:30.151907 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:30.171453 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:30.175771 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:30.177564 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:30.180981 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:30.249891 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:03:30.266961 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:30.274235 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:30.275795 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:30.276826 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.279366 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.281116 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:30.281831 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:30.281873 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:30.303678 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:30.431255 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:30.432536 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.434142 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:30.434463 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:30.438295 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:30.438836 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:30.529783 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:30.532456 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:30.533249 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:30.535740 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:30.536551 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:30.540226 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.540275 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:30.576699 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:30.618798 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:30.623795 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:30.624460 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.626297 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.629491 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:30.633612 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:30.671380 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:03:30.672060 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:30.738456 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:30.739917 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.742432 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.743418 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:30.745496 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:30.756094 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:30.838853 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:30.860744 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:30.862934 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:30.878186 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:30.880503 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:30.887172 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.892522 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:30.903568 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.912138 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.912972 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:30.920178 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 161 similar messages]
I20240503 14:03:30.921870 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:30.931192 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:30.946130 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:31.024969 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.032181 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:31.032907 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.036412 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:31.037556 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:31.063561 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:31.072094 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:03:31.077111 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:31.079129 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:31.084106 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:31.088450 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:31.088482 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:31.088502 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:31.104956 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.108561 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:31.187152 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:31.205456 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:31.207476 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.215446 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:31.216202 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.215435 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:31.217730 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:31.320535 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:03:31.322103 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:31.336889 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:03:31.363842 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.364877 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:31.365929 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:31.368413 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:31.371004 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.386394 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:31.482941 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:31.497247 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:31.499933 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:31.501499 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.506649 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:31.508487 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:31.514321 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:31.538832 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.569983 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:31.574774 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:31.575277 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:31.577950 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.578431 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.580940 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:31.585682 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:31.678248 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:31.681005 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.681005 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:31.681222 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:31.682492 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.685460 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:31.752768 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:31.779013 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:31.783761 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:31.784452 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.786243 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:31.794220 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:31.803798 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:31.851738 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:31.854318 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:03:31.885239 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:31.901801 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:31.902292 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:31.905627 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.906819 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.924455 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.929276 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:31.935091 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:31.936353 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:31.941989 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:31.944912 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:31.945013 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:31.950749 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:31.960014 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:31.964011 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:31.992184 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:32.068233 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:32.086184 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:32.092278 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.094331 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.096378 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:32.096412 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:32.098927 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:32.110180 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:32.112105 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:32.119443 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.128863 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.133318 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:32.133697 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:32.160226 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:32.161259 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:32.206311 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:03:32.232636 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:03:32.273301 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:32.365068 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:03:32.393643 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:03:32.396054 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:03:32.408378 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:03:32.422143 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:32.459818 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:32.461392 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:32.461867 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240503 14:03:32.463387 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:32.469841 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.476920 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.482712 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:32.482993 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:32.488754 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:32.507493 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:32.508117 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.511184 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:32.516289 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:32.530388 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.618532 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.620472 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:32.625074 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:32.625490 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:32.625663 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:32.637006 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:32.651449 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:32.651607 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.661379 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:32.662109 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:32.675158 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.677500 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.678303 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:32.709884 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:32.770184 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:32.773005 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.775522 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:32.777223 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:32.780725 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:32.791162 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:32.794451 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:32.845875 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:32.875532 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:32.877365 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.880599 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:32.881225 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:32.885010 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:32.892954 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:32.896061 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:03:32.906373 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:32.999328 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:33.006321 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:33.015568 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.022280 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:33.022747 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.031809 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:33.048429 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.054006 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:33.064489 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:33.066560 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.066637 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:33.069036 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:33.075594 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:33.137185 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:03:33.152467 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:33.166482 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:33.166961 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:33.175379 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.176138 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:33.177758 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:33.196453 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:33.198155 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:33.207903 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:33.323746 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:33.347306 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:33.347601 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:33.348215 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:33.351015 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.363176 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.378660 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:33.469480 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.471414 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:33.471908 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:33.483644 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:33.488992 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 14:03:33.500905 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:33.510485 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:33.523861 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:33.524973 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:33.526230 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.527163 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:33.531242 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.547981 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:33.644526 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:33.646760 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:33.647965 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.659557 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.659701 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:33.663070 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:33.668224 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:33.671844 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:33.671857 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.676571 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:33.680913 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:33.681172 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:33.692179 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:33.705209 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:33.730782 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:33.758019 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:33.771477 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:33.809890 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:33.813186 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.813410 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:33.819473 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:33.824718 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.825821 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:33.920094 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:03:33.922673 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:33.923578 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:33.924162 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:33.926968 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:33.931311 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:33.948719 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:33.954646 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:34.015378 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:34.018760 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.019299 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.020417 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:34.024763 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:34.032761 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:34.060624 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:03:34.119139 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:34.123363 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:34.124490 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.125983 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.126344 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:34.129040 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:34.165627 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:03:34.227161 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:34.227882 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:34.248952 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.249893 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:34.256531 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:34.256872 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:34.260584 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:34.265451 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:34.268396 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:34.276242 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:03:34.310724 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:34.358682 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:34.359977 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:34.363204 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:34.364809 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.378779 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:34.384814 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:34.401739 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:34.410032 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:34.469518 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:34.473387 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:34.474942 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.478457 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:34.480815 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:34.498108 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:34.499100 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:34.531401 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.535308 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.540582 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:34.544013 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:34.544797 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:34.555392 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:34.625042 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:34.632426 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:34.633141 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.640808 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:34.641045 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.641856 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:34.651371 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:34.655763 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:34.700439 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:34.700464 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:34.700599 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.702821 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:34.703483 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.704432 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:34.705454 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:34.809309 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:34.810706 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:34.813699 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:34.814702 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.815138 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.822041 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:34.824487 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:34.831342 30640 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:03:34.831465 30640 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:03:34.831843 30640 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:03:34.833102 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:34.833217 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:34.833271 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:03:34.833480 30640 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:03:34.848619 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:03:34.868510 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:03:34.869760 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:34.947384 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:34.947608 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:34.950943 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:34.955652 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:34.956321 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:34.956933 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:34.963649 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:35.060298 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:35.070676 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:35.073407 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:35.073501 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.073767 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.075045 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:35.095300 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:35.098479 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:35.153002 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:35.154152 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:35.156152 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:35.156877 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.160477 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:35.161331 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:35.163177 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.171298 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:35.229995 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:35.267186 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:35.285249 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.285477 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:35.291399 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:35.293804 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.295495 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:35.308434 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:35.317482 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:35.356638 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:35.360458 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:35.416328 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.420203 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.421501 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:35.422119 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:35.432858 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:35.450793 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:35.460759 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:35.462291 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:35.463238 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:35.465140 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.478776 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:35.482420 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:35.490483 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:35.496551 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:35.564407 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:35.575749 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.576269 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:35.576387 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:35.576421 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.579566 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:35.635362 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:35.637355 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.642063 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.643496 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:35.648545 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:35.649240 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:35.663312 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:03:35.673557 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:35.736845 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:35.738085 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:35.740180 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.740214 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:35.741720 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:35.743489 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.744135 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:35.761723 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:35.899567 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:35.899814 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:35.901940 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:35.902899 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:35.903986 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:35.918229 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:35.927192 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240503 14:03:35.928040 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:36.016813 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:36.021073 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:36.028113 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.031986 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.035524 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:36.055330 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:36.081758 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:36.089747 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.090242 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:36.090597 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:36.100090 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.107843 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:36.119300 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:36.188603 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:36.189267 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.192422 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:36.195327 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:36.197731 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:36.215760 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:36.215924 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:36.245494 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:03:36.282927 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:36.303184 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:36.307060 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:36.310638 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.319615 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.323341 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:36.328986 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:36.387934 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:03:36.391813 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:03:36.391898 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:36.394680 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:36.399479 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:36.402670 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.406710 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.439368 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:36.447743 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:36.485808 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:36.489017 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:36.492714 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:36.498203 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.500162 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.508755 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:36.526502 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:36.549739 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:36.556294 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:36.556846 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.562892 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:36.576110 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:36.579018 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:36.586568 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.662464 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:36.664073 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:36.667447 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.668939 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.669889 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:36.691673 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:36.722857 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:36.725627 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:36.816218 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:36.875517 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:36.878356 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:36.880038 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:36.884089 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:36.887900 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.892591 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:36.903669 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:36.931169 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:03:36.952744 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:36.993227 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:37.001880 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:37.006230 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:37.013653 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.015336 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:37.032348 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:37.034827 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:37.037212 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:37.048314 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:37.049706 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:37.052222 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:37.054179 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:37.063813 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:37.067893 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:37.070977 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:37.108714 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:37.112316 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:37.152514 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:37.157052 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:37.160202 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:37.163378 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.166599 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:37.168007 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:37.168447 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:37.245327 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:37.265173 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:37.266850 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:37.269318 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.270725 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:37.273092 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.284433 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:37.335645 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:37.383387 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:37.387770 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.398022 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:37.398653 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:37.403573 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:37.419016 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:37.419302 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:37.427199 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:37.440380 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.440454 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.441934 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:37.445142 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:37.477972 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:37.489215 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:37.551285 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:37.559437 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:37.560415 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.562286 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.567992 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:37.568513 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:37.580448 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:37.604723 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:37.611111 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:37.611110 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.611537 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.613054 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:37.615758 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:37.636193 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:37.697129 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 14:03:37.697721 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:37.725617 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:37.748708 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:37.837260 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:37.912464 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:37.916885 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.918705 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.921264 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:37.923668 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:37.928505 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:37.931742 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:37.938233 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:37.977928 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.979418 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:37.980340 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:37.987877 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:37.988818 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:37.991362 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:38.035418 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:38.075641 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.076156 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:38.078585 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:38.080585 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.083438 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:38.088295 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:38.160706 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:38.162027 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:38.162480 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.164755 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.166712 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:38.177073 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:38.189316 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:38.195506 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:38.202631 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:38.253571 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:38.271310 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:38.271569 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.272567 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.275166 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:38.277438 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:38.278769 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:38.375519 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:38.382004 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:38.387660 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:38.387861 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:38.388105 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:38.389117 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.396334 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:38.402628 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:38.462203 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:38.463724 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:38.463770 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.473661 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:38.473928 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.476037 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:38.556829 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:03:38.563648 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:38.568426 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.571322 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:38.575582 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:38.578348 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:38.587759 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.610487 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:38.610714 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:38.612687 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.624015 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.633560 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:38.634559 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:38.743769 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:38.769248 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:38.821789 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:38.842192 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:03:38.910349 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:38.912365 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:38.925005 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:38.925210 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:38.935405 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.944836 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:38.955770 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.957787 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:38.958504 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:38.969355 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:03:38.974268 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.979286 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:38.980331 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:38.995061 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:38.995817 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:38.996155 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:38.996277 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:03:39.060863 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:39.100508 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:39.107725 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:39.109680 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.111351 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.117772 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:39.135649 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:39.152632 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:39.156694 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 14:03:39.161341 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.161506 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:39.162302 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.164405 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:39.173923 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:39.259851 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:39.262238 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.265478 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.271363 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:39.275401 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:39.281011 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:39.298671 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:39.371528 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:39.373343 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.376322 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:39.376499 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:39.378576 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.396363 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:39.441870 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:03:39.444067 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:39.466797 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:39.495226 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:39.498618 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.502818 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:39.520198 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:39.522277 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:39.522599 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:39.534076 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:39.535384 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:39.544214 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.547639 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:39.552538 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:39.554080 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:39.556959 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.571535 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:39.579608 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:03:39.608107 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:39.650331 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:39.651108 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:39.654320 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:39.655941 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:39.663789 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:39.667635 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:39.732164 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:39.742049 31036 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:03:39.742156 31036 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:03:39.742619 31036 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:03:39.743057 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:03:39.743206 12036 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:03:39.743304 12036 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:03:39.743477 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:39.744062 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:39.744130 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:03:39.744366 31036 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:03:39.769063 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:39.861680 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:39.865818 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:39.869477 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:39.872411 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:39.873241 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:39.878715 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:39.880563 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:39.886023 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:39.898151 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:03:39.907128 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:39.952383 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:39.978555 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:39.988400 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:39.996326 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:39.997459 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.002458 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.004238 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:40.035176 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:40.045262 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:40.048218 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.048724 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.061625 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:40.062831 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:40.072448 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:40.154830 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:40.160681 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:40.176891 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:40.177688 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.179620 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:40.182898 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.193631 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:40.198371 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:40.200685 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:40.221974 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.222489 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:40.229545 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:40.231786 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:40.258180 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:40.274425 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:40.330260 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:40.334553 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:40.334573 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:40.334867 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:40.342006 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:40.356876 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.366417 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:40.370775 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:03:40.430670 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:40.452296 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:40.452533 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:40.453521 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:40.455943 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:40.459501 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:40.464509 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:40.464712 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.479614 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:40.536036 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:40.549741 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:40.549788 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:40.551129 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.552474 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.553622 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:40.563475 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:40.657991 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.659057 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:40.660091 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:40.664916 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:40.674963 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.675669 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:40.744239 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:40.772603 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.777349 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:40.777760 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:40.777918 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.779407 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:40.789736 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:40.809207 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:40.880060 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:40.880331 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:40.882843 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:40.888273 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:40.890323 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:40.890910 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:40.912830 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:40.926297 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:40.977378 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 161 similar messages]
I20240503 14:03:40.986594 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:40.991124 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:40.992776 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:41.003537 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.014642 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.030655 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:41.055297 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:41.069021 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.069568 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:41.070595 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.073786 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:41.074012 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:41.114271 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:41.156539 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:03:41.169746 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:41.176450 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:41.177971 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:41.182554 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:41.187911 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.196596 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:41.209692 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.238826 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:41.242139 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.242272 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:41.244594 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.255069 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:41.267499 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:41.271564 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:41.277035 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:41.331943 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:03:41.357699 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:41.358047 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:41.359267 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.359869 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.361162 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:41.365367 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:41.379563 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:41.478328 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:41.533388 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:41.534987 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.539875 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:41.545641 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.559448 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:41.559722 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:41.606158 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:03:41.644838 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:41.656550 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:41.659621 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.659773 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:41.660701 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.665113 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:41.671022 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:41.674243 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:41.726603 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:41.726900 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.733479 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:41.737983 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.739161 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:41.739823 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:41.792595 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:41.837226 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:41.837509 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:41.839864 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.843127 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.844053 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:41.855684 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:41.861899 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:41.892112 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:41.911168 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:41.958642 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:41.960443 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:41.967453 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:41.967706 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:41.968338 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:41.978282 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:42.015523 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:03:42.020654 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:42.032325 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:42.037945 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:42.039268 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.039860 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:42.045043 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.057164 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:42.067700 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:42.074275 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:42.124886 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:42.126972 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:42.129448 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:42.135020 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:42.146839 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:42.149544 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.158530 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:42.196522 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:42.197826 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:42.200176 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:42.205523 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:42.211891 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.219766 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:42.233484 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:42.252853 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:42.282994 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:42.283949 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.285369 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.293032 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:42.300172 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:42.302129 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:42.320421 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:42.338761 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:42.411549 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:42.525004 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.527330 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.527400 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:42.528550 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:42.531464 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:42.541069 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:42.635695 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:42.640175 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:42.644822 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:42.652266 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:42.658360 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:42.660768 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.665333 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.673980 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:42.679925 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:42.689646 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:42.690065 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.693692 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.697919 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:42.708351 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:42.735847 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 14:03:42.736660 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:42.766330 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:03:42.785797 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:42.801610 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:42.801707 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:42.802839 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.803062 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:42.812516 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:42.817759 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:42.828153 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:42.880612 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:42.903821 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:42.912475 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:42.913583 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:42.914140 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.915323 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:42.925436 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:42.928026 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:43.002986 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:43.010250 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:43.037616 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:43.050316 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.051642 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.056354 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:43.065362 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:43.067153 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:43.078421 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.092384 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:43.094944 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.097167 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:43.098701 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:43.106686 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:43.142943 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:43.219383 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:43.220676 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.224323 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:43.225800 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:43.240234 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:43.240376 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:43.251987 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:43.254145 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.265761 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.274230 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:43.274807 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.296280 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:43.384900 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:43.388697 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:43.514729 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:43.559573 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:43.569454 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:43.577396 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:43.585254 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.586608 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:43.600798 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.613721 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:43.625464 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:43.629837 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.632557 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:43.647696 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:43.648097 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:43.667898 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:43.671797 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:43.695797 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:03:43.717949 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:43.737257 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:43.739873 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.745048 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.749518 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:43.751531 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:43.766120 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:43.769732 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:43.776823 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:03:43.781338 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:43.799139 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:43.806370 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.806712 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:43.807765 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:03:43.807981 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:43.822664 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:43.822970 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:43.825168 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:43.909294 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:43.909931 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:43.910666 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.911372 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:43.916865 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:43.939081 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:43.995150 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:44.023825 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.026932 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:44.026952 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.028537 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:44.030470 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:44.046319 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:44.126273 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:44.129180 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:44.134439 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:44.138329 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.140117 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.162905 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:44.171177 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:03:44.185472 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:44.186542 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.186558 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.188509 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:44.198561 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:44.210311 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:44.220357 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:44.270566 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:44.282671 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:44.298938 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:44.299304 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:44.303213 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:44.306113 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.308362 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:44.311808 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:44.317341 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:44.346414 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:44.503170 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:44.517707 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:44.519431 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.519876 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:44.521458 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:44.522817 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.535446 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:44.536067 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:03:44.536430 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:44.628001 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:44.630725 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:44.644496 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:44.645084 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.652611 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:44.662454 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:44.666354 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:44.673923 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.685807 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:44.686107 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:44.686424 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:44.688328 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:44.691547 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:44.701547 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.712747 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:44.713151 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:03:44.714066 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:44.795055 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:44.797547 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.800434 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:44.809469 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:44.815310 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.815343 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:44.849093 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:03:44.849179 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:03:44.865442 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:44.874151 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:44.875470 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:44.877074 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:44.879602 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.884203 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.888406 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:44.903206 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:44.989786 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:44.991248 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:44.995338 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:45.000650 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.007544 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:45.008951 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:45.026536 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:45.124081 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:45.127099 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:45.131855 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:45.134176 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:45.140616 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:45.143342 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:45.146548 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.149407 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.151219 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:45.173264 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:45.190070 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:45.191939 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:45.199872 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:45.227620 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:03:45.298115 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:45.298552 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:45.299192 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:45.309710 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:45.313817 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.317312 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:45.328264 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:45.389058 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:45.458530 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:45.459988 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:45.464823 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:45.465770 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:45.468161 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:45.468412 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.469039 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:45.478188 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:45.525918 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:45.574944 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:45.575835 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:45.577287 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:45.577929 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:45.578194 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.583141 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.603700 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:45.643213 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:45.652851 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:45.655472 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.663751 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:45.664834 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.665580 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:45.669857 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:45.751399 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:45.752476 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.755877 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:45.765734 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:45.767525 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:45.769289 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:45.795795 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:45.819999 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:45.822430 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:45.832448 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.838514 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:45.846040 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:45.846819 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.904685 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:45.906410 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:45.907250 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:45.908278 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:45.914794 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:45.924885 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:45.967522 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:45.989436 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 159 similar messages]
I20240503 14:03:46.013885 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:46.016031 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.016153 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.016888 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:46.027335 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:46.044332 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:46.064651 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:03:46.081151 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:03:46.083825 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:46.094749 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:46.146986 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:46.149567 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.150226 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:46.151407 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:46.153941 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:46.157816 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:46.169138 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:46.203691 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:46.257261 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:46.258121 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.258950 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:46.262002 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.263327 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:46.268059 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:46.269373 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:46.269866 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:46.356941 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:46.358412 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:46.358521 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.359140 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:46.361480 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.380200 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:46.428143 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:03:46.470419 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:46.472545 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.473255 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:46.484704 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:46.485169 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.495149 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:46.504300 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:46.536908 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:46.588927 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.592425 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:46.606300 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.606648 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:46.608326 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:46.610241 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:46.638957 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:46.645030 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.649287 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:46.649701 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:46.651010 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:46.680327 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:46.759259 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:46.763401 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:46.763856 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:46.766636 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.769471 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.776499 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:46.785748 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:46.808012 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:46.827924 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:46.829355 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:46.830947 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.830997 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.831815 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:46.833173 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:46.917078 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:46.940760 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:46.942888 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.943078 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:46.943732 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:46.947450 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:46.947563 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:46.956265 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:46.966621 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:03:46.968673 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:03:47.030654 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:03:47.046802 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:03:47.053087 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:47.089236 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.090946 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:47.093187 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:47.094539 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.096654 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:47.106977 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:47.109838 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:47.188951 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:47.197800 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:47.200053 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.201151 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:47.201433 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:47.203420 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:47.225203 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:47.228507 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:47.230126 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:47.300362 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:47.303289 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:47.303645 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.313102 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:47.313105 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:47.321549 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:47.372390 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:47.391130 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:03:47.409983 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:47.411633 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:47.416594 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.422938 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:47.426875 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:47.429200 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:47.429612 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.529589 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:47.531181 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:47.536092 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:47.540132 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.542275 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:47.563021 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:47.581629 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.593184 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:47.594107 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:47.595118 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:47.599846 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.601639 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.607535 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:47.619741 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:47.634063 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:47.710281 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:47.722095 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:47.726202 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:47.727900 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.746580 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.747668 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:47.751044 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:47.756227 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:03:47.756291 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:47.770009 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.775379 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:47.778530 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:47.783231 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:47.783499 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:47.823139 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:47.885641 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:47.889385 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:47.890270 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.899638 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:47.901887 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:47.902515 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:48.013545 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:03:48.014170 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:03:48.028707 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:48.035007 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:48.056470 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:48.059607 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:48.060715 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:48.063846 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:48.071530 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.084569 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:48.145931 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:03:48.160212 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:48.172122 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:48.173676 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:48.179946 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:48.187085 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.188257 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:48.190433 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:48.212101 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:48.233350 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:48.249694 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.249846 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.255203 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:48.255328 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:48.257071 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:48.276026 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:48.359520 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:48.360708 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.362136 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:48.372063 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:48.374934 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.376389 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:48.429215 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:48.474161 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:48.480182 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:48.480919 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.481527 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:48.486495 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.487990 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:48.492965 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:48.495945 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:48.503549 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:48.547746 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:48.549594 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.559885 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.560575 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:48.567955 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:48.584113 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:48.606655 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:48.661103 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:48.661955 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:48.675155 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:48.676364 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.676618 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:48.676815 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:48.687867 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:48.718317 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:48.723163 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.730073 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:48.733307 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:48.740494 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:48.742918 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.743521 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:48.792294 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:03:48.819765 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:48.821573 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:48.824211 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:48.829145 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:48.832808 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:48.834362 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:48.835037 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 14:03:48.838481 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:48.898901 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:49.020620 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:03:49.050333 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:49.051697 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.055371 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.056565 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:49.062635 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:49.075456 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:49.148262 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:49.166119 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:49.172812 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:49.175369 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.175732 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.176072 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:49.177317 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:49.211262 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:49.226094 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:49.226351 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:49.228236 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.230392 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.239373 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:49.245199 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:49.272688 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:49.296593 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:49.323335 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:03:49.326668 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:49.329720 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.333775 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.335566 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:49.337852 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:49.349511 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:49.442565 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:49.446296 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.448221 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.448315 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:49.455931 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:49.456954 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:49.497176 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:49.542328 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:49.546042 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:49.553838 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:49.559743 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:49.563416 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.569391 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:49.580857 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:49.587762 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:49.590749 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:49.592991 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:49.596357 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:49.613642 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:49.617323 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:49.623011 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:49.626726 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.626890 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:49.642429 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:49.713043 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:49.714659 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.716182 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:49.721016 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:49.721698 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:49.738119 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:49.740348 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:49.753613 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:49.789059 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:49.790717 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:49.791352 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:49.792937 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:49.798308 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:49.799618 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:49.818603 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:49.829128 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:03:49.854887 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:49.929818 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:49.989974 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:50.026352 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:50.030874 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.031118 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:50.039001 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:50.042112 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.047804 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:50.051741 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:50.078809 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:50.146382 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:50.148252 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:50.151094 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:50.152031 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.152536 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.167956 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:50.174469 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:50.177712 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:50.183033 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:50.204355 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.207410 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.212205 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:50.249387 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:50.298895 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:50.301399 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.302284 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:50.303395 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:50.309623 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:50.311404 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:50.323601 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:50.405661 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:50.408303 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.408725 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:50.410755 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:50.413141 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:50.416471 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:50.429553 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:50.447866 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:50.468025 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:50.489322 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:50.522544 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.525408 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:50.529359 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:50.536852 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:50.539059 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.539698 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:50.543756 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:50.548146 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.550851 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:50.552547 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:50.572605 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.575592 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:50.577991 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:50.603542 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:50.679396 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.689646 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:50.693423 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.694355 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:50.698627 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:50.700636 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:50.704160 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:50.708151 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:50.714772 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.719607 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:50.721489 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:50.750886 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:50.754087 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:50.806317 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:50.858152 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:50.866456 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:50.921208 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:50.946733 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:51.011992 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240503 14:03:51.019690 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:51.070322 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.075110 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:51.075582 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:51.080479 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:51.080715 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:51.080727 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.083760 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.084993 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:51.086081 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:51.095062 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.108279 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:51.115525 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:51.195591 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:51.215057 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:51.220780 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:51.230588 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:51.232086 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:51.232175 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.241091 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.252944 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:51.261937 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:51.273921 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:51.276047 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:51.280557 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.291188 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:51.293130 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:51.293349 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:51.305639 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:51.379199 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.380403 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:51.380906 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:51.384933 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.387387 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:51.408059 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:51.482609 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:51.483808 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:51.486335 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:51.489583 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.491232 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:51.493932 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:51.501144 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:51.519976 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:51.579483 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:51.583225 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.587002 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:51.588696 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:03:51.590337 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:51.592307 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.593732 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:51.651468 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:51.651511 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:51.651705 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.654217 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.673103 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:51.676141 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:51.700790 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:03:51.721259 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:51.732908 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.733799 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:51.734755 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:51.736129 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.754361 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:51.759193 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:51.775434 1554 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:03:51.775548 1554 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:03:51.775914 1554 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:03:51.776878 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:51.776991 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:51.777107 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:03:51.777246 1554 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:03:51.796154 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:51.825467 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:03:51.960023 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:51.962096 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:51.962189 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:51.962747 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:51.962756 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:51.967782 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:52.032132 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:03:52.067020 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:52.083354 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:52.089315 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:52.089877 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:52.091567 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.100123 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:52.101368 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:52.103380 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.104105 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:52.105423 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.107367 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:52.114758 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:52.115638 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:52.117018 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:52.127624 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:52.136883 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:52.147037 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:52.162285 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:52.176115 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:52.177027 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:52.269402 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:52.270898 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.271016 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:52.274914 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:52.275516 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.287663 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:52.302248 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:52.311614 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:52.315151 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:52.322690 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.324210 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.339749 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:52.343189 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:52.384406 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:03:52.397891 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:03:52.426123 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:52.462388 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:52.463418 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.464022 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.470232 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:52.472088 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:52.483685 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:52.499471 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:52.596122 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:52.612249 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.615873 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:52.627844 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.628350 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:52.629532 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:52.642133 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:52.712400 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:52.742278 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:52.795924 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:52.796075 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:52.797677 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.798277 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.798585 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:52.802503 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:52.907280 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:52.909350 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:52.915266 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:52.916786 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:52.917493 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:52.917635 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:52.961872 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:53.009938 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:53.012143 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.013000 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.019901 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:53.023288 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:53.038831 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:53.055564 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:03:53.056164 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:53.101401 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:53.122046 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:53.125546 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:53.125797 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.130275 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:53.132640 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.140260 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:53.234392 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:53.241051 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:53.245241 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:53.245853 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:53.255971 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:53.259182 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.260864 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:53.264209 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:53.265326 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:53.272665 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:53.272727 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:53.275496 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:53.281524 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:53.290964 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:03:53.300105 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:53.310060 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:53.313153 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:53.349054 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:53.356194 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:53.408093 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:53.410272 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:53.423403 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:53.434132 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.435889 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:53.444268 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.446672 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.451754 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.452400 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:53.454820 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:53.479524 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:53.481277 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:53.561136 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.564117 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:53.565771 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:53.567097 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:53.577303 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.580982 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:53.637717 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:53.638464 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:03:53.721968 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:53.768636 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.769418 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:53.772315 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:53.772922 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.782346 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:53.796000 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:53.845209 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:53.884560 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:53.887176 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:53.893082 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:53.896875 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.904443 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:53.910497 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:53.927269 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:53.927351 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:53.928346 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:53.929443 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:03:53.932780 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:53.940997 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:53.943944 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:53.962352 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:53.969033 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:54.026146 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.026796 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:54.027120 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:54.028082 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:54.030570 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.035390 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:54.045409 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:03:54.116616 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:54.130080 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:54.130607 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:54.132759 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:54.135316 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.136291 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.147434 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:54.155380 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:54.254302 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:03:54.256074 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:54.260007 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:54.262184 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:54.264717 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:54.266712 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.266933 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:54.271178 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.274899 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:54.277835 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.296314 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:54.298523 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:54.306902 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.324047 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:54.331276 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:54.351629 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:54.436034 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.438783 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:54.439944 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.441355 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:54.452059 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:54.462472 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.463734 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:54.473171 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:54.474298 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:54.476856 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:54.477178 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.491058 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:54.510993 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:03:54.512467 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:03:54.552404 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 14:03:54.574169 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:03:54.596958 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:54.661049 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:03:54.667606 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:54.736212 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:03:54.779119 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:03:54.786998 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:54.806021 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:54.807575 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:54.808957 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.809778 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:54.811208 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:54.820971 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:54.821677 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:54.823439 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:54.828179 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:54.829249 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:54.837199 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.851565 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:54.858455 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:54.879761 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.970825 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:54.971073 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:54.980458 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:54.985925 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:54.986241 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:54.989825 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:55.005739 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.011164 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:55.021132 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:55.025966 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:55.028499 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.031283 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.047588 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:55.048200 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:55.049865 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:55.129453 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:55.131150 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:55.134747 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.135903 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:55.137199 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:55.138413 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:55.154340 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:55.158535 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:55.190217 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:55.234565 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:55.236274 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:55.237327 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:55.241529 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.253044 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:55.259543 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:55.261691 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:55.363857 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:55.368628 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.373473 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:55.379287 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.392021 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:55.393306 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:55.396976 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:55.407123 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:55.424583 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.455803 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.459004 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:55.460283 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:55.522367 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.525461 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.528978 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:55.529815 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:55.530048 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:55.531430 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:03:55.658751 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:03:55.674710 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:55.693930 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:03:55.708352 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:55.717237 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:03:55.730165 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:55.741382 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.742449 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:55.744887 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.747547 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:55.748600 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:55.748849 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:55.799927 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:55.847507 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:55.848773 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.849602 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:55.860589 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:55.860895 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:55.862289 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:55.886421 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:55.887465 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:55.890190 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:55.902878 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.902951 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:55.903712 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:55.913915 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:56.019930 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:56.028656 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:56.029564 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:56.030812 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:56.032502 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:03:56.036181 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 160 similar messages]
I20240503 14:03:56.038307 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.039532 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.042214 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.042537 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:56.053864 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:56.058777 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.059521 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:56.073597 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:56.144896 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:56.162938 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:56.166713 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.166814 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.173287 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:56.185459 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:56.187891 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:56.193897 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:56.203336 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:56.251914 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:56.270817 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:56.276620 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.277343 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.280550 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:56.286377 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:56.288839 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:03:56.291563 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:56.370028 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:56.371400 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:56.372745 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.373719 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:56.388357 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:56.389918 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:56.399773 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:56.405200 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:03:56.478893 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:56.480566 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.480613 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:56.482574 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:56.496308 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:56.510515 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.586390 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.586783 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:56.588629 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:56.590412 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.590636 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:56.591534 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:56.689321 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:56.689389 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:56.692328 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:56.693871 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:56.695739 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:56.695855 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.701862 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.712450 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:56.732718 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:03:56.745827 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:56.763252 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:56.810467 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:56.822187 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:56.823824 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.823997 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.828832 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:56.830545 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.832500 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:56.833248 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:56.839812 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.847015 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:56.849931 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:56.852327 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:56.864405 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:56.968818 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:56.972174 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:56.977741 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:56.978713 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:56.988191 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:57.000690 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:57.004180 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:57.007189 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:57.010594 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.010818 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:57.015872 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.028801 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:57.047681 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:57.051887 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:57.096020 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:57.127189 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:57.130250 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:57.131083 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:57.132692 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.133248 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.134738 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:57.172546 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:03:57.245723 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:03:57.248855 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:57.273823 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:03:57.340005 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:57.340281 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:57.342787 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:57.343356 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.351393 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:57.352119 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:57.354862 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:57.369791 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:57.384466 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:03:57.444422 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:57.466013 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:57.467124 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:57.469249 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:57.471289 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.471897 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:57.477360 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:57.486311 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:57.487468 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:57.496847 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:57.503676 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.505726 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.514546 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:57.526448 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:57.540771 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:57.555287 8798 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:03:57.555403 8798 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:03:57.555971 8798 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:03:57.556743 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:03:57.556852 12036 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:03:57.556910 12036 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:03:57.557201 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:57.557921 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:57.557989 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:03:57.558210 8798 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:03:57.594841 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:03:57.603385 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:57.609548 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:57.611589 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:57.615157 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.630076 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:57.631708 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:57.635368 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.716341 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:57.717378 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:57.720363 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.729133 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:57.730310 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.745715 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:57.825268 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:57.837401 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:57.837608 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.838371 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:57.841094 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:57.856570 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:57.858268 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:57.872375 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.872771 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:57.875661 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.893091 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:57.896339 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:57.904404 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:57.959035 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:57.984891 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:58.003430 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:58.004042 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.010624 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:58.011288 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:58.036140 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.043576 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:58.046629 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:58.047506 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:58.047749 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.048244 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:58.049850 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:58.061945 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.072841 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:58.168906 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:03:58.169481 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:03:58.235823 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:03:58.283561 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:03:58.287319 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:58.323652 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:03:58.361631 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:58.371610 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.372251 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.372444 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:58.385381 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:58.388118 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:58.395326 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:58.429226 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:58.430706 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:58.432343 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:58.436153 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:58.441216 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.449748 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:58.468710 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:58.509806 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:58.524780 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:58.527587 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:58.534889 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.536563 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:58.538197 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.538466 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:58.586570 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:58.586901 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:58.587522 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.596829 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.605067 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:58.613552 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:58.628059 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:03:58.661605 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:03:58.692523 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:58.695839 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:58.697458 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:58.705178 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.709486 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.718250 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:58.791637 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:58.797886 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:58.802345 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.804381 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:58.804603 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.811446 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:58.824743 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:58.846647 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:03:58.864879 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:03:58.866199 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:58.884169 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:58.887784 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:58.888496 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:58.894935 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.895191 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.896358 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:58.944347 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:03:58.958578 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.960583 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:58.961963 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:58.965446 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:58.987730 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:58.990657 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:03:59.015188 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:59.034330 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.034969 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:59.038228 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.047322 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:59.047698 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:59.064910 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:59.078161 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:03:59.197126 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:59.262578 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:03:59.275223 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:03:59.278008 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:59.282434 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:59.283386 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:59.284093 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.295941 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:59.298170 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:59.325837 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:03:59.393972 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:03:59.396008 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:59.406345 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:59.408865 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.420819 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.422185 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:59.423342 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:59.436398 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.439458 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:59.443912 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.448889 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:59.469003 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:59.470486 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:59.490003 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:03:59.504514 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:03:59.529708 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:03:59.563714 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:59.565399 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:59.572480 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:59.579366 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:59.580806 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.585278 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:59.588088 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:59.594588 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:59.594946 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:03:59.595479 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:59.595605 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:59.603701 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:03:59.608646 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:59.612640 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:03:59.704100 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:03:59.716437 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.717876 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:59.719321 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:59.722900 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:59.736565 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.736634 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:59.769021 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:03:59.783792 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:03:59.812198 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:03:59.827137 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:59.827683 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:59.830529 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.833832 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:03:59.844882 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:03:59.848786 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.857431 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:03:59.924891 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:03:59.935829 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:03:59.937122 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:03:59.938019 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:03:59.940234 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:03:59.952922 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:03:59.960330 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:00.002478 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:00.010522 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:04:00.038990 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:00.039983 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:00.040598 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.041180 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:00.043540 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:00.048689 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:04:00.053979 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:00.074405 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:04:00.104249 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:00.151634 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.152036 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:00.152521 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.152966 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:00.153049 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:00.153440 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:00.202741 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:04:00.267407 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:00.270607 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:00.275528 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:00.278142 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.282193 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.286274 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:00.382571 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:00.385152 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:00.393886 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:00.395300 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:00.396412 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.396772 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:00.408820 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.427124 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:00.439517 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:00.439961 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:00.443727 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:04:00.444213 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:00.453114 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.457437 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:00.524049 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:00.524088 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:00.571223 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:00.579733 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:00.580839 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:00.581343 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.604477 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:00.605733 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:00.608630 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.609926 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:00.614040 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.617584 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.635592 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:00.636482 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:00.720736 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:04:00.738602 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:04:00.739948 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:00.740375 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:00.746371 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:00.752082 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:00.753675 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.755956 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.756953 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:00.784533 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:00.806663 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:04:00.845446 1554 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:04:00.845571 1554 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:04:00.845963 1554 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:04:00.846930 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:00.847023 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:00.847105 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:04:00.847254 1554 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:04:00.878168 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:04:00.938567 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.940090 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:00.944020 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:00.945183 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:00.955122 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:00.965173 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:00.978135 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:01.055219 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 159 similar messages]
I20240503 14:04:01.074098 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:01.075882 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:01.076442 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.078465 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:01.090888 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:01.104136 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:01.111116 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:01.116398 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.125811 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:01.131448 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.132151 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:01.147516 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:01.148128 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:01.219071 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:01.237520 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:01.239909 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:01.241151 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.241489 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.244828 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:01.253362 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:01.260941 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:04:01.320817 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:01.349650 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.352700 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:01.356664 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.363473 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:01.367417 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:01.379400 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:01.441589 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:04:01.465520 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:01.467705 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:01.474476 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:01.474736 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.475003 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.511171 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:01.520813 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:01.524776 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:01.530385 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.530957 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:01.533898 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:01.542770 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:01.561936 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:01.637100 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:01.653398 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:01.657965 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:01.661727 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.671085 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:01.673801 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:01.676223 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.682387 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:01.687374 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:01.698943 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:01.703042 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:01.703349 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:01.724656 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:01.725324 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:04:01.728746 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:04:01.731164 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:01.806617 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:01.824348 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:04:01.864197 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:04:01.909298 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:02.028856 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:02.032217 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.032338 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.039129 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:02.044502 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:02.053700 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:02.092396 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:02.095412 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.100841 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:02.101907 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:02.103600 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:02.103860 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:02.107765 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:02.135805 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:04:02.162770 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:02.174829 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:02.205492 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.205595 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:02.207067 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:02.208048 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.208644 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:02.210494 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:02.220417 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:02.222137 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:02.252547 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:04:02.255497 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:02.277540 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:02.282837 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:02.284798 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:02.287384 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.287634 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.289113 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:02.392405 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:02.394546 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.399430 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:02.400488 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.402704 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:02.426836 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:02.438241 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:02.522264 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.522468 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:02.526436 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:02.530732 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.537125 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:02.547540 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:02.549194 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:02.590183 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:02.592375 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.595121 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:02.596761 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:02.604779 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.620009 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:02.634991 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:04:02.643421 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:02.664290 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:02.698700 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:04:02.707870 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:02.710196 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:02.713094 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:02.715576 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.718919 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.726234 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:02.727171 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:02.744882 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:02.747411 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:02.768457 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:02.787182 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:02.790177 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:02.794683 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:02.823302 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:04:03.056494 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:03.060971 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:03.066354 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:03.072322 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.075938 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.083039 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:03.085655 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:03.108426 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:03.139099 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:03.140055 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.144542 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:03.147595 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:03.152552 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:03.167415 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:03.174060 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:04:03.217172 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:03.232004 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:03.234755 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.237185 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:03.251704 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.266289 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:03.268566 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:03.273550 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:04:03.274500 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:03.303999 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:03.307205 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:03.311499 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.314216 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.314343 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:03.319177 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:03.327948 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:03.357580 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:03.416689 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:03.416704 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.419327 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:03.427768 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:03.431540 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.433626 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:03.448976 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:03.539342 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:03.539629 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:04:03.539937 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.547621 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:03.548460 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:03.560206 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.561300 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:03.601418 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:03.606127 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:03.671631 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:03.677841 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:03.682915 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.689947 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.690848 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:03.712285 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:03.713647 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:03.728538 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:03.728659 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:03.733384 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:03.738394 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:04:03.739023 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.740059 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:03.764603 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:03.843355 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:03.845894 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.847620 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:03.848892 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:03.851127 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:03.851938 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:03.869518 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:04.039456 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:04:04.054221 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:04.062531 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:04.064764 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:04.066892 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.067314 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.067934 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:04.077667 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:04.096483 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:04.114251 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:04:04.136380 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:04.176940 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:04.183853 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.186509 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:04.191239 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.197196 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:04.207814 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:04.214985 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:04.242513 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:04.242513 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:04.256819 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:04.257290 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:04.276579 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.279543 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:04.310091 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:04.318634 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:04:04.353916 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:04.356081 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:04.358410 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:04.360685 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.361320 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.371670 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:04.372670 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:04.379740 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:04.380364 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:04.435330 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:04.439395 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:04.442657 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:04.444259 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.444461 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:04.444859 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:04.494750 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:04.500308 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:04.517650 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:04.546700 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:04.548046 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:04.555405 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.556027 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:04.558008 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.564563 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:04.646471 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:04.668607 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.668787 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:04.671792 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:04.672595 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.679368 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:04.683477 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:04.735443 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:04:04.754688 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:04.759485 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.764590 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:04.766603 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:04.777061 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:04.777101 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:04.815541 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:04.884263 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:04.885643 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:04.888931 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:04.895392 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:04.897352 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:04.901739 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:04.903782 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:04.955143 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:04:04.955852 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:04.988063 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:04.998212 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:05.016798 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:05.019515 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.022146 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.022585 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:05.022908 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:05.036983 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:05.088994 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:05.135542 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.139402 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:05.139518 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:05.147536 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.151131 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:05.163714 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:05.192194 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:05.237823 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:04:05.246598 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:05.248924 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:05.250754 8798 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:04:05.250861 8798 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:04:05.251400 8798 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
W20240503 14:04:05.253621 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:05.253952 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:04:05.254076 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:04:05.254168 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
I20240503 14:04:05.256256 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.256659 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:05.257282 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:05.257382 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:04:05.257519 8798 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:04:05.261358 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:05.271353 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:05.314196 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:05.316094 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:05.318655 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.331077 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:05.340656 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:05.343664 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:05.354439 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:05.412528 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:05.415740 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.417330 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:05.421644 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:05.421931 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:05.435657 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.448820 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:05.479746 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:05.508450 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:05.510484 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:05.514035 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:05.519357 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:04:05.531399 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.531692 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:05.545159 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:05.584064 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:04:05.631475 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:05.631722 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:05.632467 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:04:05.639421 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:05.640460 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:05.648479 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:05.652245 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.663901 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:05.736949 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:05.741394 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:05.765081 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:05.771993 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:05.772495 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.773461 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:05.773602 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.778111 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:05.822356 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:04:05.883008 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:05.885329 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:05.885969 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.891566 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:05.894008 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.904945 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:05.992673 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:05.993793 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.996837 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:05.996858 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:06.006163 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:06.006573 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:06.092795 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:06.094090 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240503 14:04:06.094839 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:06.110620 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:06.110946 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:06.112430 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.113638 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:06.121397 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:06.126071 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:06.129395 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:06.130607 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.219208 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:06.220439 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:06.220875 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:06.225831 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.225899 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.257212 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:06.266016 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:06.286489 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:06.290802 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.291386 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:06.293413 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.301792 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:06.306533 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:06.330153 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:06.340898 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:04:06.418589 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:06.425055 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.439180 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:06.440896 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:06.443804 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:06.446630 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.453755 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:06.459915 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.462568 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:06.472575 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.473351 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:06.488054 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:06.522344 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:04:06.580792 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:06.583209 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:06.586302 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:06.586807 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.591910 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:06.592659 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.599362 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:06.656700 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:06.681543 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:06.733616 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:06.740238 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:04:06.781972 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:06.783174 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.789551 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.791821 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:06.798274 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:06.802503 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:06.873538 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:06.914208 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.916869 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:06.919977 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:06.927834 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:06.927933 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.928052 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.930531 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:06.935206 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:06.943493 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:06.964726 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:06.971372 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:06.984771 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:06.992838 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:06.997180 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:07.005383 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:07.025799 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:07.071781 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:07.074066 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:07.079437 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.082326 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:07.082983 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.089432 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:07.125128 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:07.177589 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:07.181047 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:07.185153 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:07.188504 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.188779 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:07.189388 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.189815 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:07.205011 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:07.205245 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:07.284147 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:07.294140 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.294976 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:07.296422 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:07.297729 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:07.301784 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:07.333024 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:07.340144 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:07.367130 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:07.367401 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.369473 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:07.374908 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.375195 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:07.380168 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:07.391738 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:07.396677 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:07.472255 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:07.473646 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:07.475042 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.483376 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.488246 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:07.497848 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:07.507566 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:07.551409 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:07.551461 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.554014 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.564077 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:07.564684 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:07.576392 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:07.693495 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:07.737978 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:07.776562 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:07.780723 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:07.796258 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.796878 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:07.799824 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.800084 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:07.820840 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:07.821205 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:07.828051 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:07.906385 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:07.914474 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.916594 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:07.919440 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:07.920181 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:07.930464 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:07.939714 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:07.959832 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:07.974356 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:07.977639 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:07.978135 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:07.979200 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:07.986665 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:07.997642 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.086707 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:08.087091 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.092478 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:08.092483 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.097046 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:08.105731 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:08.120254 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:08.171813 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:08.180647 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:08.199059 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.203773 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:08.205925 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.207340 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:08.211397 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:08.236269 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:08.240190 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:08.315363 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:08.315483 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:08.317167 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:08.320889 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:08.330639 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.331732 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:08.340955 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:08.359628 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 14:04:08.380472 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:08.383008 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.389453 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:08.391414 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:08.394063 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.394728 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:08.424361 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:08.478359 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:08.512787 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:08.515592 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:08.517884 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.519399 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:08.520076 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:08.522019 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:08.522460 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.531524 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:08.536428 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.537662 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:08.554217 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:08.554596 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:08.554615 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:08.661213 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:08.686957 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:08.738766 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:08.854709 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:08.861948 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:08.874330 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:08.879818 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:08.887192 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.895365 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:08.907519 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:08.915117 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:08.915807 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:08.917088 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:08.922480 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.928913 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:08.933300 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:08.952275 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:08.959388 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:09.066293 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.067343 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.067354 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:09.068806 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:09.068987 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:09.080812 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:09.104945 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.107455 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:09.114629 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:09.115829 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.123970 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:09.130755 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:09.131340 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:09.140395 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:04:09.228617 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:09.228893 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.229003 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:09.235049 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.236503 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:09.236912 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:09.249919 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:09.292431 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:09.337658 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.337878 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:09.341410 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:09.344492 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 14:04:09.345078 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:09.350446 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:09.355715 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:09.365140 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:09.372885 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:09.407171 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:09.446931 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:09.465090 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:09.475514 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.479097 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:09.479398 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:09.480746 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.481328 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.490660 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:09.499886 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.507921 12279 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:09.510488 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:09.518913 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:09.520893 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:09.544042 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:09.628908 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.636878 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:09.637892 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:09.637905 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.641733 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:09.641940 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:09.654310 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:09.661808 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:09.694607 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:09.773900 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:09.822180 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:09.822794 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:09.834370 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:09.841531 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.842820 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:09.850180 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:09.853698 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:09.855960 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:09.913440 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:09.955256 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:09.959306 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:09.960073 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.960227 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:09.964088 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:09.964339 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:09.977025 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:09.989894 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:09.991820 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:10.006283 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:04:10.021206 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.022346 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.024118 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:10.024482 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:10.028525 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:10.044781 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:10.140265 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:10.144887 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:10.144949 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.157392 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:10.158124 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.161273 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.169672 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.173257 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:10.185292 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:10.194727 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:10.206373 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:10.209244 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:10.226825 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:04:10.292546 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:10.295550 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:10.299182 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:10.299937 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.301789 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:10.302465 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:10.305009 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:10.307173 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:10.311975 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:10.388301 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:10.401491 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:10.408665 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:10.412941 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:10.416363 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:10.416388 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.418237 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:10.438191 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:10.457901 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:10.522303 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:10.522572 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.527994 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:10.528754 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:10.529939 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.535786 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:10.581283 15295 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:04:10.581427 15295 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:04:10.581909 15295 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:04:10.582636 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:04:10.582764 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:04:10.582839 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:04:10.582906 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:10.583107 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:10.583174 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:04:10.583333 15295 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:04:10.596072 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:10.632444 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:10.635751 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:10.635958 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:10.638319 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:10.645440 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.646438 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:10.671821 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:10.681833 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:10.731680 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:10.745457 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:10.746969 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:10.748227 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:10.763705 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.764470 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:10.772668 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:10.772780 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:10.840265 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:10.858338 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:10.862802 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.864013 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:10.867441 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:10.877285 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:10.884202 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.977929 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:10.978152 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:10.983405 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:10.993958 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:11.002254 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:11.018817 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.032039 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:11.034875 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:11.035197 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:11.045401 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:11.045444 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.048329 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.051308 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:11.056586 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:11.104753 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 157 similar messages]
I20240503 14:04:11.160485 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:11.168047 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.169463 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:11.190052 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:11.190768 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:11.202453 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.202571 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:11.218132 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.219889 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:11.220628 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:11.221385 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:11.222769 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:11.230605 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:11.241865 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:11.251914 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:11.267469 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:11.343839 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.344954 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.346061 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:11.350976 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:11.351416 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:11.361471 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:11.467056 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:11.515125 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:04:11.531559 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.535717 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:11.537407 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:11.539201 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:11.547816 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:11.561486 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:11.565977 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:11.631127 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:11.653569 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:11.663195 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:11.664248 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.671605 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.675386 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:11.680033 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:11.681898 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:11.716457 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:11.725178 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.729643 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:11.731751 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.736485 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:11.740175 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:11.789602 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:11.837495 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:11.840451 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:11.844682 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.845435 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:11.856871 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:11.857617 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:11.865581 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:11.949347 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:11.954849 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.962406 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:11.962915 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:11.965082 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:11.965943 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:11.966975 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:04:11.980482 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:12.049850 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:12.055104 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.057394 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:12.060825 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.064193 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:12.090436 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:12.128341 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:12.129380 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:12.130801 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:12.137835 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.147578 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.147828 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:12.155736 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:12.181821 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:04:12.232095 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:04:12.237372 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:12.240710 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:12.243816 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:12.246896 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:12.247056 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:12.247583 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.256561 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:12.260308 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:12.286563 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:12.296965 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:12.296994 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:12.300288 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:12.302829 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:12.313717 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:12.317525 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:12.318634 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:12.361903 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:04:12.376829 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:12.428010 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:12.623848 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:12.625567 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:12.629431 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:12.646921 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.651578 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:12.656023 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:12.666042 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.667603 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:12.668872 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.677551 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:12.686342 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.714089 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:12.749778 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:04:12.774905 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:12.804569 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:12.813210 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.813643 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:12.815519 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:12.816495 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:12.831085 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:12.851511 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:12.860283 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:12.860786 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:04:12.862749 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:12.867812 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.869495 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:12.871445 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:12.872335 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:12.875669 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:04:12.883196 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:12.888971 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:12.902652 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:12.971025 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:12.975697 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:12.978674 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.986371 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:12.987084 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:12.987085 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:12.995645 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:13.067001 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:13.092337 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.092391 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:13.092954 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:13.098580 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:13.107473 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:13.111205 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.210417 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:13.214613 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:13.220335 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:04:13.220909 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:13.230823 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:13.232425 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.236351 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:13.238696 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:13.244141 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:13.258688 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:13.263325 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.271764 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.275821 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.278242 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:13.321801 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:13.371476 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.374743 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:13.379093 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:13.383399 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:13.383641 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.386454 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:13.398456 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:13.411510 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:04:13.414297 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:13.414664 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:13.517120 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:13.578114 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.580228 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:13.581511 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:13.588090 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.588351 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:13.589751 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:13.607326 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:04:13.648545 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:13.653704 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:13.708317 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:13.708340 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:13.712636 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:13.713487 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:13.713778 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.723222 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.723634 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.742691 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.744967 13886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:13.745195 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:13.761636 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:13.762704 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:13.792186 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:13.875455 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:13.880998 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:13.883401 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.891402 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:13.892859 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:13.898519 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.899700 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:13.916630 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:13.917966 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:13.923703 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:13.930680 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:13.955358 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:13.956787 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:13.959759 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:14.038965 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.039340 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:14.040222 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:14.043480 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:04:14.044402 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.047416 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:14.060691 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:14.112175 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:14.144428 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:14.149320 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:14.152662 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.154146 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:14.154954 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:14.166692 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:04:14.167845 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:14.255559 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.259681 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:14.262377 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:14.262872 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:14.263850 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:14.264443 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.365834 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:14.368999 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:14.370329 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:14.375609 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.379599 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:14.388206 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:14.389946 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 14:04:14.390566 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:14.404482 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:04:14.436625 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:14.476351 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:14.477675 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:14.478076 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.481500 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:14.492599 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.508744 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:14.589184 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:14.590160 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:14.593967 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.594235 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:14.594945 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:14.599642 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:14.605266 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:14.635488 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:14.665302 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:14.667160 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:14.704981 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:04:14.711917 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:14.718223 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:14.727710 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:14.733131 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.734714 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.736799 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:14.739869 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:14.748558 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.752637 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:14.769353 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:14.769917 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:14.777334 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:14.789160 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:14.790654 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:14.890940 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:14.894798 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.899210 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:14.901141 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:14.920115 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.921274 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:14.928773 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:14.940869 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.944177 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:14.949641 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:14.959848 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:14.960799 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:14.963511 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:14.964987 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:14.972667 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:04:14.997334 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:15.026285 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:04:15.057431 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:15.058319 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.060634 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:15.061182 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:15.062069 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.065136 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:15.151392 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:15.159426 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:04:15.194589 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:15.254978 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.256026 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:15.261013 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:15.264256 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:15.268553 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.269583 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:15.334980 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:15.379002 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.382683 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:15.389348 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:15.390147 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.396136 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:15.398115 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:15.399363 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:15.403962 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.406723 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:15.406862 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:15.407058 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:04:15.415527 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:04:15.422798 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:15.427358 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:15.470664 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:15.474304 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:15.533430 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:15.534472 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:15.540063 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:04:15.543011 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.545365 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:15.545691 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:15.550189 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.550554 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:15.551025 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:15.600229 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:15.670142 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:15.670416 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:15.675297 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:15.675858 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:15.684839 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.687191 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:15.695832 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:15.746496 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:15.782725 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:15.786619 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:15.788748 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:15.800314 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.814520 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.819093 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:15.825644 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:15.830935 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:15.831552 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:15.845944 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:15.850965 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:04:15.856709 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:15.858884 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:04:15.860703 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.955220 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:15.958159 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:15.964236 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.965018 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:15.966658 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:15.975926 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:16.025177 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:16.026291 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.027894 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:16.040864 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:16.042148 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.043268 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:16.075520 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:16.106717 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240503 14:04:16.140810 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:16.218106 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:16.229559 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:16.260634 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:16.297231 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:04:16.301677 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.302935 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:16.303911 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.304646 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:16.306581 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:04:16.309144 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:16.323398 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:16.402262 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:16.406911 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:16.408120 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.412452 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.412590 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:16.431075 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:16.481245 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:16.485666 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.490976 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:16.494031 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.497994 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:16.515811 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:16.562362 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.568912 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.569886 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:16.571736 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:16.571990 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:16.591655 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:16.659992 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:16.675184 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:04:16.678042 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:16.678151 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:16.680266 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.682108 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:16.683710 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:16.684536 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:16.700495 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:16.783555 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:16.801376 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:16.802218 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:16.806248 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:04:16.816972 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.818500 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:16.819869 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:16.837181 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:16.839911 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:04:16.853264 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.859086 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:16.863118 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:16.879315 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:16.884093 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.887988 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:16.991343 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:16.992645 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:16.992838 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:16.993485 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:17.006779 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:17.009948 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:17.047554 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:17.048704 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:17.051090 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:17.051494 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:17.057284 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:17.079898 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:17.082517 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:17.138141 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:17.170020 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:17.176446 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:17.178471 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:17.196615 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:17.233953 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:17.248432 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:17.251286 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:17.359067 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:17.369103 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:17.369939 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:17.370813 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:17.371446 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:17.372696 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:17.389117 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:17.393522 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:04:17.448405 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:17.449450 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:17.452159 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:17.454493 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:17.465811 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:17.472234 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:17.477671 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:17.518618 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:04:17.549782 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:17.550324 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:17.559804 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:17.561892 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:17.566052 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:17.583796 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:17.621296 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:17.624431 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:17.625226 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:17.630226 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:17.637382 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:17.638957 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:17.651997 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:17.695977 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:17.725606 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:17.726284 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:17.728399 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:17.730718 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:17.732137 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:17.735388 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:17.789645 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:04:17.831342 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:17.835162 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:17.836589 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:17.839236 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:17.840831 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:17.845634 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:17.848728 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:17.874701 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:17.892338 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:04:17.918032 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:17.919067 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:17.951397 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:04:17.963336 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:17.963811 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:17.963862 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:17.964932 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:17.974169 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:17.994489 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:17.995067 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:17.997833 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.008380 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:18.017872 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.030086 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:18.044008 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:18.088891 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:18.097095 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:18.126538 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:18.129870 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.131326 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.133451 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:18.141954 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:18.148995 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:18.201133 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:18.273619 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:04:18.334867 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:18.334914 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:18.336133 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.344640 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:18.348975 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:18.351080 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:18.400713 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:18.439400 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:04:18.440076 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:18.457224 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:18.460505 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.467137 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:18.472867 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.476743 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:18.502127 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:18.503407 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:18.506243 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.509382 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:18.522179 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:18.522279 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:18.530360 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:18.544147 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:18.628424 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:18.636126 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:18.644331 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.644726 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:18.652006 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.653303 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:18.656244 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.668778 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:18.677943 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.684144 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:18.686647 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:18.692276 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:18.695304 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:18.719604 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:18.791025 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:18.792059 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:18.799358 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.799844 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:18.802171 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.802894 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:18.844982 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:18.904208 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:18.910846 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:18.911763 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:18.916818 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:18.920538 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:18.922601 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:18.928619 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:18.932441 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:19.019562 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:19.019582 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:19.025825 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:19.028431 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:19.028761 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.029476 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:19.044692 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:19.121320 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:19.135649 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:19.138463 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:19.139755 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:19.139916 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.153759 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.165026 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:19.196167 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:19.218801 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:19.240842 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:19.243567 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:19.244118 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.247520 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.247758 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:19.248926 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:19.311725 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:04:19.356467 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:19.357463 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:19.358430 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.363359 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.371511 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:19.372538 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:19.393325 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:04:19.460723 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:04:19.481951 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:19.490582 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.492561 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:19.492589 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:19.492842 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.501822 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:19.508874 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:19.513561 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:19.531824 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:19.538425 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:19.549623 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.551630 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:19.552402 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:19.553988 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:19.621843 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:19.643548 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:19.644244 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.648136 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:19.655254 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.659741 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:19.665077 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:04:19.676728 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:19.684193 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:19.691712 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:04:19.703794 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:19.710028 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:19.715927 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:19.718060 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.723714 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:19.728142 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:19.728593 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:19.742108 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:19.747264 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:04:19.749249 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:19.771618 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:19.829169 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:19.831910 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.833153 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:19.834231 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.835829 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:19.847452 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:19.896275 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:19.906365 16145 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:04:19.906507 16145 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:04:19.906903 16145 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:04:19.907737 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:19.907938 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:19.908020 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:04:19.908138 16145 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:04:19.915871 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:19.946337 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:19.986716 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:19.988296 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:19.991564 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:19.999997 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:20.003192 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:20.006273 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.112804 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:20.114822 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:20.115520 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.122702 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:20.125885 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:20.129853 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.192548 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:20.195107 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:20.195298 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.198137 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.209409 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:20.232779 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:20.245147 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240503 14:04:20.260557 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:04:20.274320 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:20.318466 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:20.320124 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:20.329069 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:20.330044 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.331251 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.339614 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:20.353601 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:20.370335 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:20.396787 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:04:20.402518 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:04:20.442672 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:20.450145 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:20.450407 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:20.460850 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:20.461354 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:20.462436 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.462785 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.485682 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:20.499771 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:20.505946 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.512532 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.517726 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:20.522460 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:20.523321 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:20.524751 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:20.620667 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:20.624819 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:20.635429 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.645078 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:20.647323 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.672319 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:20.685801 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:20.686345 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.689496 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:20.692203 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:20.698649 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:20.700217 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:20.700407 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:20.756983 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:20.800879 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:20.800879 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.804008 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:20.804798 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:20.807659 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:20.811614 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:20.811962 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:20.874118 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:20.971421 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:20.976848 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:20.976848 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:20.982826 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:20.988113 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:20.989902 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:20.995738 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:20.998750 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:21.101059 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:21.106298 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:21.106989 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:21.109314 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:21.115401 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.118396 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:21.127754 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 159 similar messages]
I20240503 14:04:21.173353 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:21.175699 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:21.177093 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:21.178444 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:21.178830 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.179100 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.292433 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:21.294404 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:21.294930 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.299149 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:21.309104 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.311607 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:21.366525 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:21.377722 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:04:21.409972 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:21.417805 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:21.421763 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:21.425217 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:21.426209 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.426828 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:21.428030 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.434118 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:21.456010 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:04:21.463826 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:21.494145 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:21.498256 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.499533 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:21.500629 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:21.501610 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.507609 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:21.533282 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:21.600436 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:21.601140 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.605139 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:21.607283 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:21.608207 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.615638 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:21.668234 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:21.671620 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:21.680985 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:21.682463 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.683414 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.683482 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:21.708894 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:04:21.709692 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:21.771337 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:21.773056 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:21.777233 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.784159 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:21.786796 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:21.788908 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.803181 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:21.842993 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:21.994066 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:21.994834 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:21.997205 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:21.998564 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.009160 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:22.009928 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:22.017971 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:22.028250 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:04:22.035496 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:22.106534 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:22.112309 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:22.115820 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.120601 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:22.121277 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:22.141845 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:22.155279 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:22.171607 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:22.172204 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:22.172981 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.176494 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:22.179036 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:22.184829 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:22.193267 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:22.247212 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:04:22.261106 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:22.279629 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:22.283931 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:22.287153 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.290306 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.292223 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:22.293202 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:22.299211 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:22.346208 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:22.391697 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:22.406987 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.410435 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:22.413450 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:22.419433 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.423910 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:22.427982 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:22.429179 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:22.472651 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:22.514091 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:22.517792 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.527459 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:22.538389 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:22.541641 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:22.542344 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:22.585098 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:22.592319 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:22.592929 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:22.593246 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:22.594343 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.596546 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.605057 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:22.613126 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:22.694299 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:22.695027 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.700672 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:22.706213 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.718618 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:22.732769 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:22.766875 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:22.768643 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.771916 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:22.773131 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:22.773934 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:22.782929 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:22.783705 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:22.857771 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:04:22.878628 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:22.942808 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:22.978041 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:23.042157 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:23.048115 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.048979 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:23.049845 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.050247 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:23.053041 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:23.061827 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:04:23.073853 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:04:23.090329 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:04:23.140161 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:23.140872 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:23.144452 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:23.145100 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.149740 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:23.157456 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:23.173839 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:23.177830 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:23.224488 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:23.227072 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.227216 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:23.228845 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:23.230173 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:23.251435 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.259140 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:23.272235 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:23.308452 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:23.308825 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:23.311594 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.312202 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.319928 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:23.331768 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:23.388468 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:23.391443 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:23.431223 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.433087 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:23.436678 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:23.438104 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.438376 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:23.447178 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:23.535523 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 14:04:23.562099 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:23.566200 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:23.569604 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.577145 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:23.577324 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:23.581483 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.589242 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:23.596092 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.597671 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:23.607771 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:23.625164 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:23.635457 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:23.658608 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:23.731006 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.731339 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:23.737347 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:23.737920 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.743424 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:23.768860 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:23.772868 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:23.778939 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:23.785439 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:23.785552 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.787724 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:23.794399 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:23.799750 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:23.801854 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:23.813722 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:23.920375 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:23.980711 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:23.993188 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:23.994766 16145 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:04:23.994912 16145 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:04:23.995362 16145 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:04:23.996357 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:23.996531 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:23.996610 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:04:23.996757 16145 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:04:24.057272 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:04:24.094714 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:24.095042 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:24.098003 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:24.104554 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.114518 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.127730 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:24.154903 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:24.159458 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:24.163559 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.169909 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:24.173597 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.190594 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:24.227345 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:04:24.269378 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:24.274567 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.279067 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:24.287896 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:24.290328 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.291424 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:24.291503 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:24.342655 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:24.343561 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:24.351713 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.357062 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:24.360924 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:24.363453 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:24.377972 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:24.398476 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 14:04:24.403718 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:24.450847 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:24.457890 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:24.458195 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:24.459147 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.462257 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.474637 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:24.492077 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:24.574051 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:24.574573 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:24.581375 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.581869 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:24.583492 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.604753 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:24.605587 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:24.659113 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:24.678085 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:04:24.685580 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:24.685910 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:24.687896 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.699206 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:24.700346 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:24.704780 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:04:24.717927 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:24.750886 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:24.751668 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:24.760000 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:24.760044 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.764681 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.768132 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:24.773159 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:24.828006 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:24.841904 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:24.842417 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:24.852929 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:24.858242 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:24.861063 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:24.863526 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:24.863526 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:24.865067 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:24.865795 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:24.955538 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:04:25.052476 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:04:25.073797 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:25.076893 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.077481 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:25.078183 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:25.080765 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.090474 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:25.110155 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:25.142525 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:25.205209 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:25.206962 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:25.210712 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.211323 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:25.214732 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:25.215422 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:25.225000 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:25.225207 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:25.225374 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:25.234899 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.237861 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.255517 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:25.274565 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:25.277616 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:25.362018 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:25.366497 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:04:25.372874 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:25.385792 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:25.387487 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:25.388861 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:25.390727 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.391842 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.393438 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:25.395756 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:25.403389 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:25.406945 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.428716 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:25.439210 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.440977 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:25.465752 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:04:25.547649 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.548429 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:25.548421 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:25.549880 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.551928 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:25.564159 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:25.565246 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:04:25.575769 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:25.602840 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:25.663625 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:25.664355 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:25.666106 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.668581 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:25.670748 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:25.679082 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:04:25.680143 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:25.703636 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:04:25.760780 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:25.761765 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.761915 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.771397 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:25.776777 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:25.785681 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:25.792893 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:25.804427 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:25.825898 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:04:25.866714 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:25.868745 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.872611 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:25.873034 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:25.873273 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:25.890995 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:25.954417 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:25.975646 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:25.977710 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:25.978576 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:25.981029 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:25.993341 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:26.003042 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.088240 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:26.091298 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:26.091701 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:26.091773 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.092815 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.103828 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:26.125627 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:26.149356 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 157 similar messages]
W20240503 14:04:26.150468 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:26.185463 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:26.189491 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:26.192057 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.197651 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:26.198158 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:26.211678 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:04:26.215656 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:26.232036 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:26.259429 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:26.259994 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:26.261423 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.265170 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.272483 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:26.273775 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:26.335893 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:04:26.342990 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:26.343912 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.344786 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:26.353248 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:26.357018 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:26.357167 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:26.367741 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.425758 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.427537 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.433434 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:26.436959 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:26.438684 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:26.451856 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:26.540066 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.541771 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:26.542138 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:26.542215 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:26.547320 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.552073 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:26.625838 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:26.646006 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:26.660904 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:26.660996 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:26.661207 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.676256 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.684366 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:26.684435 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:26.739046 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:26.742067 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:26.756129 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:04:26.773773 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:26.776316 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:26.776487 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.783583 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:26.786572 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:26.795782 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:26.804479 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:04:26.831493 19842 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:04:26.831605 19842 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:04:26.832095 19842 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:04:26.832585 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:04:26.832742 12036 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:04:26.832829 12036 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:04:26.834021 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:26.834101 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:26.834161 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:04:26.834514 19842 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:04:26.863497 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:26.865466 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:26.876310 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:26.877324 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.879261 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:26.879619 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:26.905404 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:04:26.974782 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:26.975395 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:26.976601 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.977079 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:26.979364 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:26.980250 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:27.035306 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:27.050166 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:27.101641 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:27.111413 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:27.116137 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.128564 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:27.129676 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:27.132617 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:27.139741 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:27.139884 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:27.149305 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:27.165280 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:27.169039 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.170295 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.172058 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:27.195107 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:27.259369 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:04:27.266985 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:27.269773 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:27.272187 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.272609 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:27.275614 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:27.289219 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.310312 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:27.319734 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:27.320029 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:27.320415 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.324749 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:27.334348 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.344156 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:27.440961 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:27.441222 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.446942 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.453536 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:27.455449 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:27.465519 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:27.488953 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:27.524977 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:27.559473 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:04:27.565899 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:27.575073 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:27.620852 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:27.627593 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:27.631209 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:27.637295 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:27.645022 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.651626 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.656517 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:27.678416 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:27.699937 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:27.700490 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:27.707175 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:27.746313 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:04:27.749791 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:27.753333 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.754200 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:27.754678 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:27.762789 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.783457 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:27.784138 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:27.790705 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:27.797492 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.810650 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:27.812964 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:27.818284 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:27.847913 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:27.872422 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:04:27.905864 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:27.908200 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:27.912076 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:27.917070 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:27.917199 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:27.932725 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:27.943542 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:27.991614 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:04:28.018435 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:28.021209 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:28.031347 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.032984 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:28.034034 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:28.054437 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:28.088263 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:28.114957 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:04:28.125532 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.126852 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:28.131764 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:28.139098 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:28.147662 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.149706 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:28.153383 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:28.178843 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:28.183722 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.186703 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:28.193948 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:28.204895 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:28.213148 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:28.232465 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:28.292623 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:04:28.302687 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:28.307031 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:28.315549 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:28.316047 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.323371 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:28.324656 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:28.330546 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:28.340055 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:28.347118 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:28.354661 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.354640 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:28.364363 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:28.374955 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:28.493067 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:28.538221 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:04:28.571146 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:04:28.579025 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:04:28.669931 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:28.676096 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.681474 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:28.692015 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:28.700871 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:28.701435 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:28.702076 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.705102 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:28.715210 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.726810 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:28.734647 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.741312 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:28.750694 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:28.757638 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:28.761322 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:28.830982 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:28.833570 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:28.833796 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.840597 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.850114 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:28.851424 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:28.863590 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:28.876843 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:28.877939 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:28.882752 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:28.891045 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.899372 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.919330 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:28.958580 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:28.977305 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:28.980382 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:28.980701 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:28.984701 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:28.989758 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:29.003360 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:29.027087 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:29.088366 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.089399 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:29.091730 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:29.096544 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:29.097549 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:29.104333 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.194363 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:29.196278 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:29.197530 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:29.206630 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:29.206938 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:29.208266 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.227640 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.260685 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.261657 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:29.262878 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:29.266698 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:29.275691 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:29.284437 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:29.311134 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:29.358048 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:29.362013 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:29.364920 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:29.365363 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.365923 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:29.369871 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:29.392468 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:29.403959 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:04:29.415364 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 14:04:29.416078 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:29.531353 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:29.532791 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.533290 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:29.539628 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.543069 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:29.544793 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:29.593677 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:29.631361 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:29.631470 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:29.635850 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:29.637362 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:29.638038 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:29.648406 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:29.649639 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.718968 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:29.720508 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:29.723317 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:29.724378 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.725714 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.732993 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:29.733511 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:29.753357 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:29.769897 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:29.793324 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:29.795802 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:29.796018 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.796499 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:29.796923 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.820744 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:29.874879 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:29.883587 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.889029 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:29.892156 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.894156 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:29.898389 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:04:29.900025 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:29.900249 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:29.929934 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:29.956486 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:29.957304 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:29.960335 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:29.962671 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.963390 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:29.963589 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:29.964274 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:30.028425 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:04:30.056192 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:30.066649 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:30.072216 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:30.074466 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.080545 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:30.086970 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:30.088188 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:30.115890 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:30.155519 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:04:30.161835 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:04:30.185644 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:30.198127 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:30.198395 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:30.199151 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.209713 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.220559 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:30.223469 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:30.276600 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:30.277978 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:30.309197 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:30.309710 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.310240 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:30.319764 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:30.321630 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:30.323482 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:30.350332 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:04:30.353802 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:30.385232 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:30.397358 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:04:30.407349 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:30.408869 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.410284 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:30.416471 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.421675 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:30.442718 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:30.481602 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:30.516633 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.517215 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:30.519001 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:30.523499 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:30.531406 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:30.537627 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:30.548871 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:30.629894 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:30.633991 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.641259 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:30.641707 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:30.646730 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:30.649549 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:30.652065 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:30.652390 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:30.656471 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:30.667383 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.667819 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:30.687682 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.696534 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:30.707026 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:30.778861 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:30.786132 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:30.788143 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:30.791231 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:30.794903 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.800644 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.801964 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:30.806955 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:30.817490 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.818519 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:30.821353 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.828863 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:30.829805 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:30.852144 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:30.870132 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:30.928671 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.929019 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:30.930392 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:30.934350 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:30.935389 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:30.937548 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:31.090859 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:04:31.123163 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:31.140446 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.140552 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:31.143801 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:31.145306 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:31.152431 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:31.153436 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:31.164567 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:31.170002 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:31.199002 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 166 similar messages]
W20240503 14:04:31.199908 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:31.250802 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:31.255681 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:31.257818 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:31.259657 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.267419 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.269482 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:31.292140 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:31.305514 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:31.316766 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:31.317144 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:31.317601 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.320801 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:31.321764 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:31.327389 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.408879 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:31.410655 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:31.411376 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:31.412190 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.425441 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.446254 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:31.522171 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.524608 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.524694 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:31.525286 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:31.529824 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:31.531412 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:31.546409 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:31.635013 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:31.635385 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:31.635453 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.638991 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.661218 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:31.665820 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:31.675204 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:31.675868 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:31.678404 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:31.679694 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:31.700064 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:31.701328 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.703038 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:31.716799 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:04:31.756728 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:31.801081 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:31.804387 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.805060 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:31.808730 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:31.817553 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:31.834138 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:31.836035 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:31.837970 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.847193 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:31.848393 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.856186 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:31.870977 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:31.888428 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:04:31.924842 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:04:32.058503 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:04:32.066434 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:32.097657 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:32.169827 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:32.173123 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:32.173597 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.176664 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:32.180080 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.183374 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:32.194769 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.207728 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:32.217132 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.220747 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:32.221642 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:32.223012 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:32.240993 16145 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:04:32.241089 16145 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:04:32.241464 16145 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:04:32.244275 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:32.244596 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:32.244650 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:04:32.244763 16145 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:04:32.245573 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:32.250020 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:04:32.252967 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:32.280108 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:32.313359 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:32.319487 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.320612 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.324090 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:32.333390 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:32.334044 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:32.348488 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:32.380722 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:32.382946 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.385107 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.385710 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:32.392406 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:32.393502 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:32.430572 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:32.487411 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:32.488444 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.491117 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.493122 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:32.494964 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:32.510629 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:32.515753 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:32.599702 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:32.602368 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:32.604718 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:32.608472 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:32.614063 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.622620 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:32.631606 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:32.632931 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:32.686869 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:32.690366 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:32.693128 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:32.693781 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:32.696787 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.697885 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:32.705971 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:32.707374 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:32.739085 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:32.755899 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:32.761600 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:32.766696 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.768014 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:32.770138 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.772681 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:32.788486 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:04:32.794761 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:32.798743 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:04:32.799189 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:32.842978 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:32.843909 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:32.847128 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:32.853997 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.854456 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:32.876931 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:32.963876 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:04:32.977327 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:04:33.060636 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:33.065620 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:33.065899 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:33.068243 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.071450 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:33.077402 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:33.088408 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:33.090106 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:33.134393 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:33.169190 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:33.172024 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:33.179811 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.187741 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:33.188446 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:33.192001 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:33.208153 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:33.224975 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:33.234411 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:33.237380 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.239563 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:33.247193 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.247628 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:33.265425 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:33.329666 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:33.335026 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:33.335736 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.336702 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.337703 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:33.359720 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:33.393576 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:33.406505 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.406524 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:33.412178 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:33.415303 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:33.434710 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:33.439976 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.518617 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:33.524104 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:33.524165 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:33.536005 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.537498 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:33.550926 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:33.617228 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 14:04:33.624784 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:33.629168 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:33.650952 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:33.652164 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:33.657306 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:33.657421 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:33.663741 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.669852 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.698642 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:33.703094 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:33.707968 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:33.713042 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.713057 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:33.714110 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:33.718220 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:33.732924 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:33.736003 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:04:33.794574 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:33.805462 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:33.805892 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:33.807214 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:33.818570 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:33.819056 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:33.833390 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:33.858007 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:34.000156 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:34.003794 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:34.003958 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.004732 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:34.006686 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.019402 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:34.043898 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:34.084410 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:34.127491 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:34.131719 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:34.138212 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:34.139955 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.141041 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.151690 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:34.154023 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:34.157673 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:34.169137 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:34.169426 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.178232 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:34.185968 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:34.186218 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:34.241071 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:34.274789 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:34.289867 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.290812 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:34.294183 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:34.303125 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.306668 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:34.320014 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:34.322901 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:34.337452 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:34.339951 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:34.350436 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.355614 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:34.367336 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:34.367491 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:34.431171 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:04:34.450719 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:34.450738 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.454617 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:34.458325 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:34.474982 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:34.475150 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:34.493902 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:34.505441 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:04:34.512508 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:34.560721 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:34.561816 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:34.566211 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.567723 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:34.568318 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:34.572242 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:34.608276 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:34.651263 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:34.654148 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:34.654633 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:34.659250 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.661341 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:34.664579 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:34.665545 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:34.696838 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:34.744916 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:34.756388 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:34.758798 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:34.760908 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:34.762724 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.775550 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.777025 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:34.871101 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:34.875406 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.876094 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:34.879658 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.885174 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:34.893077 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:34.979974 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:34.980640 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:34.982632 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:34.982683 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:34.994174 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:34.997402 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:35.008450 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:35.057199 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:04:35.062588 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:35.076069 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:35.095168 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.095393 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:35.096343 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:35.099527 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.102494 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:35.110138 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:35.116874 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:35.134624 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:35.138887 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:35.142930 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:35.147866 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:35.153761 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.163920 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:35.169560 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:35.174901 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:35.192749 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:35.214715 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:35.226716 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:35.228001 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:35.257197 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.265460 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:35.269392 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.270613 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:35.272439 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:35.287695 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:35.289675 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.296111 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:35.299412 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:35.307024 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:35.308344 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.315739 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:35.329953 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:35.332098 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:35.413169 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:35.417032 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:35.419881 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:35.422060 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:35.431881 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:04:35.439744 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.448282 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:35.527251 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:35.577996 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:35.583122 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:35.609907 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:35.610764 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:35.610771 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:35.613952 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:35.623487 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:35.628222 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:35.629091 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:35.632131 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:35.668715 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:35.729934 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:35.730871 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:35.747051 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:35.750983 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.753934 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:35.754194 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:35.761957 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:35.784261 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:35.795046 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:35.796960 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:35.799422 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.802518 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:35.803313 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.905007 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:35.905385 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:35.906324 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:35.907146 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:35.908228 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:35.916287 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:35.932404 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:36.008921 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:36.016664 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.018266 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:36.019591 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:36.020022 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:36.031574 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.054805 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:36.121477 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:36.125614 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.126200 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:36.127969 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.130940 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:36.137750 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:36.137964 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:36.156183 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:36.181635 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:36.191380 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:36.194046 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:36.195551 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:36.203645 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 163 similar messages]
I20240503 14:04:36.204386 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:36.209452 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:36.214388 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:36.215473 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.296929 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:36.301524 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:36.303342 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.305260 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:36.306438 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:36.313385 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:36.329047 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:36.357235 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:36.360651 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:36.369151 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.371076 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.373646 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:36.374182 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:36.563078 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:04:36.571110 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:36.575531 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:36.630414 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:36.632017 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:36.635123 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:36.636207 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.644305 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.655026 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:36.680013 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:04:36.721949 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:36.747272 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:36.750883 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.759387 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:36.761042 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:36.763936 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:36.769973 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:36.786510 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:36.793255 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:36.805526 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:36.806458 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.811182 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:36.819497 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:36.822382 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:36.823354 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.904245 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:36.906311 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:36.908571 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:36.915357 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.916262 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:36.917438 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:36.926303 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:04:36.972271 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:36.979979 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:37.011162 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:37.011732 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.011997 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:37.026669 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:37.031576 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:37.036015 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.116415 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:37.119386 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.130654 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:37.131198 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:37.133726 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.138275 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:37.162351 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:37.162629 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:37.165635 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:37.168092 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.171397 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:37.189092 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:37.197201 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:04:37.198134 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:37.229413 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:37.289340 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:37.297302 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.303741 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:37.307608 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:37.307957 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:37.308343 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.323478 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.324481 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:37.327131 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:37.339416 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.341131 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:37.348770 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:37.361413 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:37.422657 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:04:37.504858 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:37.561229 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:04:37.604081 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:37.635561 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:37.647128 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:37.659987 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:37.663637 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.667371 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:37.669106 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:37.671264 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:37.671869 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:37.673116 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:37.681769 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:37.687579 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.689909 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.701756 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:37.705619 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:37.708487 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:37.730563 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:37.775606 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:37.795807 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:04:37.807428 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:37.811565 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:37.812094 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:37.822468 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:37.823763 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:37.829733 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:37.831404 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:37.840965 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:37.844234 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.844962 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:37.851517 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:37.853114 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:37.853266 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:37.860026 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.861322 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.871125 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:37.882213 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:04:37.925720 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:37.970867 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:37.974213 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:37.978417 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:37.979370 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:37.987000 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:37.994750 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.078701 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.079269 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:38.081614 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:38.082935 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:38.083425 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.083478 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:38.100587 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:04:38.117178 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:38.174795 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:38.177131 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.179639 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:38.182595 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:38.183367 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.187808 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:38.224342 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:38.235061 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.237118 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:38.237872 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:38.241170 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.241909 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:38.253665 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:38.261515 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:38.344447 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:38.345206 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:38.346814 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.347230 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:38.351867 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.355739 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:38.392838 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:38.470608 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:38.508296 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:38.520634 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:38.548921 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:38.551767 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.555125 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.560626 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:38.560726 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:38.583545 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:38.616912 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:38.664763 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:38.664899 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:38.666189 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240503 14:04:38.667737 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.672557 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.690042 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:38.707846 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:38.715723 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:38.724874 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:38.727670 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:38.729629 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.738349 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.756455 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:38.806830 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:38.848470 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:38.849123 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.856734 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:38.859047 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:38.870640 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.883731 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:38.885811 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:38.896302 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:38.896528 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.904137 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:38.907912 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:38.917910 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:39.002754 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:39.011103 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:39.015431 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.028255 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:39.028306 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:39.030781 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.031709 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:39.049033 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:39.117960 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.119308 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:39.121691 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:39.124053 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:39.127518 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:39.131707 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.132454 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:39.143711 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:39.210727 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:39.212157 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.212275 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:39.220762 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:39.227300 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:39.230095 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:39.236485 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:39.244130 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.324492 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:39.324940 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.327220 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:39.327392 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:39.330008 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.342619 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:39.344317 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:39.383772 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:39.422181 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:39.424153 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:39.424827 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.426311 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:39.437175 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:39.438326 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.439570 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:39.456446 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:39.512900 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:39.525478 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.525740 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:39.527437 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:39.529989 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.530342 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:39.543725 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:39.634507 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.640162 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:39.640409 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:39.640972 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:39.643436 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:39.655659 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:04:39.665501 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.685390 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:39.686514 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:39.687239 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.688369 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:39.690707 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:39.695824 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:04:39.706741 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:39.737648 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:39.756908 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:39.786828 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:39.792176 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.792950 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:39.793244 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:39.808075 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:39.817062 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:39.818490 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:39.819584 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:39.820766 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:39.825392 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.832104 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.836537 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:39.837895 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:39.861155 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:39.922940 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:39.943655 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:39.943804 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.944907 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:39.945816 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:39.953630 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:39.954449 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:39.963368 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:39.972822 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:04:40.054517 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:40.102741 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:40.117985 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:40.149487 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:40.150965 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:40.151829 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:40.152168 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:40.152823 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.154181 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:40.179272 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:40.214879 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:40.243403 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:04:40.257665 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:40.262810 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:40.268904 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:40.270171 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:40.274967 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.286281 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:40.288107 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:40.288801 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.294756 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.301195 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:40.307664 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:40.317713 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:40.318071 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.338771 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:40.407867 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:40.424329 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.424758 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:40.426465 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:40.429615 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.430174 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:40.445195 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:40.463333 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:04:40.512439 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:40.533598 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:40.534525 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:40.537467 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.540021 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.549369 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:40.557644 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:40.605002 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:40.633967 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:40.638813 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:40.641337 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:40.642748 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:40.647603 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.648802 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.650595 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:40.672945 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:40.707391 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:40.710703 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:40.717481 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.719542 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:40.724382 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:40.740207 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:40.766999 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:40.783288 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:40.786681 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:40.786947 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:40.787541 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.788327 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.790740 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:40.799790 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:40.833640 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:40.862550 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:40.865260 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:40.867375 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:40.871891 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:40.877043 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:40.895480 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:40.904091 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:41.071390 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:41.104130 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:41.105562 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:41.107787 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:41.109488 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:41.121117 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:41.133630 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:41.169091 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:41.203276 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:41.210992 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:41.211390 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:41.213551 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:41.225525 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240503 14:04:41.227260 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:41.227715 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:41.228041 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:41.268851 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:41.269897 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:41.270184 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:41.275028 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:41.278285 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:41.287984 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:41.306609 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:41.367717 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:41.371083 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:41.371315 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:41.375617 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:41.380865 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:41.382272 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:41.384672 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:41.468813 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:41.482956 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:41.484117 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:41.489558 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:41.494810 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:41.499136 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:41.509447 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:41.549264 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:41.586308 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:41.588549 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:41.589979 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:41.591617 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:41.602557 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:41.608024 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:41.618772 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:41.645089 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:41.645165 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:41.652004 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:41.655192 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:41.658911 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:41.659328 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:41.671586 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:41.754227 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:41.754751 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:41.758776 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:41.760713 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:41.766456 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:41.776611 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:41.780511 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:41.796945 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:41.801894 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:41.802408 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:41.803855 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:41.806061 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:41.809250 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:41.815121 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:41.817726 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:41.964063 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:42.107793 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:42.113354 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:42.126552 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:42.127122 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:42.129166 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:42.132160 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.142740 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.143424 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:42.144308 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:42.144526 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:42.145108 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:42.155797 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.171401 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:42.175351 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:42.181280 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:42.193933 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:04:42.279623 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:42.281193 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.283080 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:42.291383 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:42.292243 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:42.298831 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:42.299387 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:42.340304 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:42.348657 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:42.349849 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:42.350289 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.350299 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.351044 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:42.352713 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:42.361456 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:42.368228 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:42.379906 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:42.391454 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:42.450356 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:42.453532 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:42.457868 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:42.458122 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:42.458670 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.461493 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:42.538344 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:42.558737 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:42.566038 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:42.566581 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.567442 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:42.568070 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.568101 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:42.641525 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.648315 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:42.652511 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:42.652721 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:42.655370 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.661636 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:42.675978 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:42.688966 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:04:42.700738 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:42.714032 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:42.717520 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:42.721472 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:42.722118 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.723349 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:42.741470 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:42.744360 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:42.783592 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:04:42.798604 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:42.800340 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:42.800733 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.807132 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:42.807873 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:42.811050 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:42.824088 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:42.829285 26640 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:04:42.829385 26640 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:04:42.829917 26640 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:04:42.830351 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:04:42.830461 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:04:42.830523 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:04:42.831022 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:42.831135 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:42.831175 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:04:42.831315 26640 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:04:42.925050 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:04:42.929158 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:04:42.936240 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:43.031798 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.032886 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:43.041352 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.043818 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:43.046422 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:43.046710 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:43.065876 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:04:43.081090 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:43.142690 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.146231 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:43.147229 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:43.147909 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:43.155917 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.155963 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:43.182366 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:43.185137 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:43.189775 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.190398 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.193327 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:43.194805 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:43.215528 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:43.231609 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:43.305485 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:43.306749 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.308678 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:43.321036 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:43.324252 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:43.330806 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:43.339251 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.346254 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:43.364159 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.367293 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.367913 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:43.378685 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:43.384372 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:43.454962 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:43.473371 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.475732 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:43.476234 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:43.484866 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:43.487114 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:43.500465 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:43.537101 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:04:43.541679 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:43.582865 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:43.583693 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.583818 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:43.591445 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:43.593796 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:43.593852 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:43.642998 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:04:43.661201 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:43.684343 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.686426 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:43.688256 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:43.693882 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:43.694801 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.709430 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:43.734108 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:04:43.738934 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:43.791263 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:43.794178 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:43.797808 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.798123 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:43.800213 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.800261 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:43.801738 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:43.889217 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.890283 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:43.896410 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:43.900707 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:43.901321 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:43.906121 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:43.909246 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.911202 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:43.982846 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:43.983639 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:43.989302 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:43.993958 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:43.998982 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.015679 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:44.062633 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:44.102592 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:44.104441 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:44.110558 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.110849 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:44.123438 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:44.125509 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.143810 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:44.146346 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:44.151927 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:44.152493 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:44.156913 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:44.165035 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:44.165704 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:44.171280 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:44.227329 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:44.277201 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.279829 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:44.283231 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:44.286487 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:44.286832 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.288663 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:44.289830 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:44.292714 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:44.296254 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:44.307511 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.318575 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.325769 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:44.332315 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:44.433211 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.434118 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.436831 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:44.441890 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:44.446098 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:44.454222 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:44.455514 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:44.481456 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:44.535828 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:04:44.615842 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:44.619176 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:44.623347 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.623840 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:44.633801 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.634243 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:44.634946 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:44.648432 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:04:44.657588 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:04:44.659739 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 14:04:44.683892 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:44.742903 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:44.747956 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:44.751925 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:44.768378 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.771363 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:44.777328 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:44.780550 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.782071 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:44.797976 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:44.801578 12146 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.801769 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:44.818301 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:44.818737 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:44.879804 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:44.907356 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.909526 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:44.912711 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:44.917414 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:44.919370 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:44.922982 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:44.938652 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:04:44.939590 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:44.991600 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:45.016383 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:45.024201 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:45.032375 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:45.032688 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:45.036459 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:45.037346 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.046299 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:45.110502 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:04:45.131155 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:45.149283 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:45.157747 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.159715 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.159715 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:45.162160 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:45.162621 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.166862 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:45.172171 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:45.173343 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:45.176555 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:45.180353 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.195209 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:45.196606 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:45.231695 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:04:45.262921 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:45.307991 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:45.312496 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.319540 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:45.319696 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.320233 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:45.349030 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:45.363682 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.364660 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:45.365525 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:45.365713 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.366096 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:45.382966 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:45.385253 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:45.434845 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:45.436795 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:45.441813 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:04:45.481060 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:04:45.483659 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:04:45.602859 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:04:45.665633 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:45.667935 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.669227 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:45.671957 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:45.672824 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.694387 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:45.730821 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:45.736091 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:45.740582 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.743935 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:45.749249 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.754487 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:45.776311 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:04:45.831172 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:45.859166 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:45.862774 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.873086 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:45.873641 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:45.887478 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.895829 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:45.912750 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:45.914186 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:45.918103 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:45.934064 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:45.936944 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:45.942669 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:45.943869 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:45.948298 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:45.948410 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:46.033346 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:46.035411 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:46.039997 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.040235 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:46.040349 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.045238 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:46.062316 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:46.081845 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:46.139096 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:46.142306 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:46.144026 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.145474 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:46.148475 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:46.150882 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:46.164260 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:46.249301 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:46.250742 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:46.261107 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.261552 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:46.272148 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.277619 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:46.288339 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:46.293983 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:46.296187 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:46.308290 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 166 similar messages]
I20240503 14:04:46.310463 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:46.319216 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:46.338665 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:46.374876 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:46.390790 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:46.415202 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:46.417433 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.420676 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.424361 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:46.432447 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:46.435866 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:46.436133 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:46.465359 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:46.481452 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:46.508780 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:46.621317 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:46.624632 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.625209 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:46.629565 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:46.635365 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:46.635875 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:46.636471 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:46.753343 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.755728 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.761361 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:46.765398 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:46.770642 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:46.771384 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.772643 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.777184 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:46.800459 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:46.808432 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:46.811532 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:46.815341 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:46.858211 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:46.901700 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:46.929774 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:46.934746 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:46.937688 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:46.939920 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.943641 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:46.952575 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:46.953560 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:46.955659 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:46.961194 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:46.963656 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:46.974094 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:46.977581 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:46.988832 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:46.993099 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.002310 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:47.055724 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:47.086807 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:47.087436 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.088119 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:47.088534 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:47.091897 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.093895 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:47.186486 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:47.188926 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:47.191385 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:47.191805 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.193809 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:47.196848 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:47.203274 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:47.234978 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:04:47.295310 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:47.296265 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:47.296275 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.308998 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:47.312261 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:47.319583 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:47.368026 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:47.386698 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:47.413290 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:47.417186 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.417547 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.418371 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:47.420459 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:47.425127 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:47.437304 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:47.472893 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:04:47.474490 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:47.524432 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:47.524926 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:47.525381 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.535692 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:47.537784 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.538007 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:47.548409 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:47.574481 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:47.626583 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:47.628175 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.628410 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:47.634755 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:47.634979 29771 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:04:47.635075 29771 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:04:47.635502 29771 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:04:47.636642 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:47.641690 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:47.645350 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:47.648048 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:47.648134 12103 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:04:47.648432 29771 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:04:47.649080 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:47.689015 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:47.746783 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:47.752938 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:47.758175 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:47.759611 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.768792 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.786751 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:47.792105 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:47.797065 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:47.798624 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:47.801692 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.804003 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:47.819959 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:47.827212 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:47.832701 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:47.876394 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:47.878331 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:47.898156 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:47.907459 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:47.932005 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.935885 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.935942 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:47.936446 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:47.950868 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:47.956002 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:47.957911 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:47.961367 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:47.974577 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:47.978808 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:47.991652 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:47.992283 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:47.992900 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.081445 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:48.083941 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.085688 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:48.086632 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:48.086899 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:48.096318 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:48.126320 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:48.285180 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:48.287752 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.294138 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:48.294505 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:48.294618 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:48.295027 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.299432 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:48.315771 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:48.394987 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.399673 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:48.399957 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:48.401859 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.401939 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:48.407586 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:48.408418 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:48.433843 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:48.435735 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:48.471820 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:48.472273 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:48.481493 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.481495 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:48.487071 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:48.493050 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:04:48.510073 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:48.558929 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:48.560629 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.562037 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:48.564458 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:48.576586 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.587679 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:48.623910 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:48.664947 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:48.665331 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.669756 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.675489 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:48.676082 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:48.677534 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:48.683965 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:48.777292 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:48.781065 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.781206 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:48.786221 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:48.788029 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.806877 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:48.816529 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:48.823773 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:04:48.824771 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:48.841814 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:48.842444 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:48.844529 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.848644 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.854923 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:48.886015 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:48.903797 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:48.928102 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:48.955348 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:48.956014 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:48.956073 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:48.957577 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:48.958735 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:48.963544 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:48.966476 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.033217 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:49.036844 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.037976 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.039132 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:49.043741 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:49.060016 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:49.084124 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:04:49.211098 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:49.266992 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:49.268497 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.268831 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.270157 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:49.270635 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:49.276402 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:49.292373 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:49.378104 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:49.378247 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:49.379768 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:49.382824 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:49.384439 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.414271 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:49.414472 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:49.416234 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:49.418100 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.431602 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.435914 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.447880 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:49.455390 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:49.461135 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:49.488194 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:04:49.501929 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:49.564836 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:49.566375 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:49.570923 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.571106 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:49.577463 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.588008 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:49.601174 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:04:49.604470 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:49.605793 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.617702 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:49.623690 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:49.625967 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:49.626003 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:49.683923 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:49.719252 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:49.720080 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:49.724996 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:49.726054 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.728758 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.730521 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:49.792867 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:04:49.799113 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:49.821437 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:49.824527 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:49.825592 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.826056 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:49.826689 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:49.833732 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:49.834232 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:49.875785 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:49.919867 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:49.934427 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:49.934491 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.939040 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:49.944859 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:49.947551 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:49.948873 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:49.980005 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:04:49.997786 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:04:50.019695 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:50.029664 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:50.029856 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:50.035954 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:50.052860 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:50.054138 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:50.055188 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.064222 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.065115 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:50.075662 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:50.089030 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:50.154026 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.154630 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:50.162051 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:50.162425 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.163269 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:50.169598 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:50.194757 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:50.234444 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:50.259814 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.260812 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:50.264658 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:50.268199 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:50.274554 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.282481 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:50.306087 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:50.380834 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:50.383656 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.385890 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:50.386026 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:50.388078 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:50.398511 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:50.398955 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.449635 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:50.451735 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.452037 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:50.454126 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:50.455878 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:50.463466 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:50.482262 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:50.537101 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:50.537894 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.538810 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.541092 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:50.547581 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:50.548775 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:50.570075 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:50.605654 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:50.611192 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:50.612322 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:50.614302 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:50.616411 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:50.618024 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:04:50.620604 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.621625 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:50.627871 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 14:04:50.701166 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.706514 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:50.706782 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:50.713033 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:50.715911 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.718606 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:50.795897 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:50.926537 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:50.928448 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:50.929126 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:50.930225 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:50.931259 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:50.941102 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:51.023773 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:51.034544 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:51.042574 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:51.042618 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.045975 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:51.052518 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:51.064872 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:51.066737 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:51.070421 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:51.083123 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.084470 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.093494 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:51.096531 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:51.103083 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:51.104039 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:51.144423 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:51.151924 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:04:51.155201 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:04:51.181658 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:51.182827 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:51.187631 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.190246 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.190373 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:51.212224 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:51.220325 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:51.240653 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:04:51.302397 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:51.304188 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.311436 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:51.316218 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:51.316522 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:51.323423 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.403281 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:51.404233 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:51.404434 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.407816 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:51.412513 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:51.415743 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 161 similar messages]
I20240503 14:04:51.421428 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:51.424851 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:51.470598 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:51.482363 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:51.490084 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:51.493474 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:51.494730 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:51.498318 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.502979 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.566774 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:51.567966 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:04:51.570225 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:51.571961 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:51.578085 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:51.579414 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.582163 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:51.613763 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:51.646925 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:51.647785 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:51.648645 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:51.653900 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:51.654721 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:51.662093 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.670868 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:51.672462 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:04:51.686549 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:51.880434 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.880452 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:51.886703 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:51.887094 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:51.887459 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:04:51.897825 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:51.916416 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.983211 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:51.986744 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:51.991050 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:51.992713 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.003394 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:52.005054 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:52.009935 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:52.061550 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:52.063314 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:52.064085 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:52.064873 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.079511 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:52.093286 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:52.109284 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:04:52.139685 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:52.140636 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:52.144053 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.144987 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:52.146692 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:52.156438 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:04:52.168349 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:52.172627 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:52.193475 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:52.250340 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:52.251899 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:52.255892 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:52.256254 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.257395 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:52.270051 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.277369 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:52.310317 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:52.335824 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:52.359017 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:52.359961 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:52.365278 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.367638 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:52.374011 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:52.376513 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.377813 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:52.433797 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:52.434641 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.439961 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.443327 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:52.447358 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:52.475672 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:52.526623 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:52.528848 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:52.533404 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.533404 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:52.535856 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:04:52.547572 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:52.553876 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:52.559310 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:52.567332 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:52.589677 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:52.601234 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:52.603066 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:52.604601 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:52.608290 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.611374 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:52.612057 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:52.627985 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:52.707532 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:52.754262 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:52.778179 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:52.796901 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:52.840965 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:52.842494 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:52.845461 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.850201 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:52.857703 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:52.859206 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.939615 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:52.940505 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:52.943933 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.945812 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:52.949025 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:52.963016 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:52.999047 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:53.006835 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.006860 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:53.019537 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:53.020161 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:53.032845 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:53.038054 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:53.099673 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:53.104041 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:53.104159 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:53.104501 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:53.115720 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.115839 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:53.121295 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.123399 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:53.168234 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:53.172894 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:53.174988 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.176128 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:53.177075 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.180976 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:53.186868 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:53.194120 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:53.279706 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:53.281543 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:53.287768 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.293660 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:53.300964 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.311317 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:53.357754 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:53.398451 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:53.398635 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.406047 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:53.409282 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.410240 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:53.420972 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:53.476228 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:53.479264 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:53.481060 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.484154 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.486979 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:53.487550 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:53.499007 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:04:53.573249 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:53.578119 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:53.579022 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.580121 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:53.591688 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:53.598563 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:53.602298 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:53.632058 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:53.638777 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:53.698120 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:04:53.707499 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:53.709256 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:53.710482 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.713824 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.730137 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:53.730273 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:53.757215 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:53.776671 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:53.816413 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:53.818848 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:53.821233 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.823310 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.824225 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:53.841589 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:53.873817 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:04:53.874922 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:53.902479 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:04:53.926275 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.933122 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:53.936481 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.937312 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:53.942548 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:53.956569 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:53.967094 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:53.969663 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:53.972556 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:53.975369 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:53.976469 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:53.977195 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:54.030776 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:54.035938 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:54.093284 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:54.100435 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:54.109937 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:54.110219 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:04:54.112973 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:54.114300 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.129858 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.131130 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:54.134853 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.142298 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:54.142606 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.159652 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:54.162401 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:54.164315 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:54.260007 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.262068 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:54.262352 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.263063 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:54.279773 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:54.287698 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:54.293725 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:54.350924 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:54.455578 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:54.467064 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:54.469427 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.469547 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:54.472447 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:54.472463 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.472901 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:54.590584 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:54.590798 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:54.593684 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:54.600411 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.602607 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.603366 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:54.606597 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.622448 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.625224 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:54.632354 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:54.632923 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:54.634305 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:54.636992 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:54.673638 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:54.718289 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:54.724256 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:04:54.740216 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.740890 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:54.741079 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:04:54.741818 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:54.742292 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.742987 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:54.754402 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:54.772411 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:54.842025 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:54.844173 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:54.844653 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:54.844755 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.847394 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.850420 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:54.862583 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:04:54.868685 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:04:54.880365 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:54.922086 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:04:54.958834 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:54.961830 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.962271 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:54.976207 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:54.991636 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:54.999745 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:55.004160 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:55.006989 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:55.017201 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:55.017647 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.017989 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.018288 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:55.032119 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:55.057024 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:55.076234 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:55.125864 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:55.136965 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:55.137569 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.141619 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.149928 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:55.160341 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:55.162744 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:55.163210 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.163690 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:55.167644 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:55.183558 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:04:55.184113 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:55.189267 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:55.196343 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:55.269692 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:55.363324 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:55.382355 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:55.436571 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:04:55.485085 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:55.486621 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:04:55.487406 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.489616 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.493908 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:55.496938 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:55.526196 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:55.528239 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:55.568948 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:55.569944 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:55.575141 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.575776 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:55.579337 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.580300 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:55.653221 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:55.667099 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:55.672467 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:55.673126 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:55.673321 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:55.674404 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.674552 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.739755 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:55.743973 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:55.746567 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.747545 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:55.749926 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.764235 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:55.813616 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:04:55.852444 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:55.853457 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.855262 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.857707 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:55.857728 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:55.858907 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:55.911693 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:55.936622 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:55.973536 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:04:55.987308 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:55.990123 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:55.990772 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:55.991670 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.006753 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:56.017231 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:56.053743 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:56.057585 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:56.058182 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:56.061582 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.064908 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:56.066244 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:56.083544 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:04:56.101576 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:56.101866 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:56.121619 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:56.130036 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:56.159000 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.162703 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:56.163486 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:56.175807 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.180416 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:56.193626 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:56.200230 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:56.202365 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:56.214435 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:56.222291 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:56.222631 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.239663 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:56.239826 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:56.362318 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:04:56.392580 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:04:56.448299 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240503 14:04:56.448961 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:56.516677 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:56.519140 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:56.525660 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:56.530831 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.538772 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:56.545464 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.575220 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:56.580044 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:56.586791 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:56.586885 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.587224 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.587289 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:56.608315 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:56.642131 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:56.659603 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:56.667979 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:56.668220 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:56.670073 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:56.671510 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:56.687170 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:56.694500 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.731853 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:56.736454 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:56.751541 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:56.752287 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:56.762023 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:56.763077 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:56.764430 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.818701 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:56.821014 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.824030 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:56.824446 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:56.831040 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.835412 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:56.851657 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:56.918970 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:56.920176 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.921555 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:56.925313 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:56.927466 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:56.931418 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:56.996580 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:04:57.003739 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:57.035533 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:57.040474 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.043288 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:57.043642 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.057418 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:57.061929 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:57.107868 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:57.108024 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:57.110069 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.117349 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.127398 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:57.129786 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:57.157120 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:57.170198 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:04:57.202772 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:57.207989 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:57.212040 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.212218 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:57.219033 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:57.228704 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.233175 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:57.266319 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:04:57.277258 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:57.333464 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:57.345680 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:57.346501 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:57.350764 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:57.350824 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:57.354116 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:04:57.355423 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.369300 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:57.450932 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:04:57.461622 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:57.463965 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:57.465329 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:57.467625 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.470664 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:57.475394 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.479559 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:57.526372 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:57.559733 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.561714 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:57.566560 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:57.572646 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.573808 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:57.589430 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:57.591744 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:57.604651 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:57.605777 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:57.611070 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.621309 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:57.623672 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.630771 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:57.657930 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:57.720621 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:04:57.738900 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:57.744117 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:57.747812 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:57.751899 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:57.761173 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:57.766875 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:57.776701 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:57.777087 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:57.781224 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:57.787403 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:57.794939 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:57.795958 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:57.802997 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.809681 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.818065 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:57.898415 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:57.900919 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:57.903106 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:57.911408 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:57.912707 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:57.915114 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:57.945351 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:04:58.025513 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:58.077471 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:58.078511 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:58.079308 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:58.083822 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.086539 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:58.088059 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.179111 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:58.180835 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:58.182754 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:58.185918 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:58.187184 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:58.191949 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.197646 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:58.205134 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:58.224067 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:04:58.233839 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:04:58.251962 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:58.256708 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:58.256822 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.267910 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:58.271627 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.272006 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:58.358682 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:58.359174 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.359620 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:58.370386 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:58.371104 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.380997 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:58.427079 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:58.431216 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:58.476406 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.477833 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:58.478679 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:58.483127 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:58.505837 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:58.506211 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:58.533356 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:04:58.547832 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:04:58.553967 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:58.554697 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.555852 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:58.558835 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:58.566852 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:58.582460 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:58.601816 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:04:58.606472 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:04:58.626541 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:58.627002 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:58.629595 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:58.639477 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.640321 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:58.652066 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:58.656484 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:58.663606 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:58.734342 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:58.737533 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:58.741010 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:58.747424 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:58.755496 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.756261 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.792903 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:58.796386 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:58.797003 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:58.799645 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:58.809242 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:58.810788 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:58.820915 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:04:58.866398 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:58.969442 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:04:59.043835 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 14:04:59.068193 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:59.073395 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.073542 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:59.074021 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:59.090629 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.094074 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:59.123773 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:04:59.150820 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.152451 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:59.158809 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:59.160228 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.167019 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:59.167496 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:04:59.176899 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:04:59.201812 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:04:59.237298 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:59.238453 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:59.239881 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:59.245499 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.256917 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.260488 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:59.261601 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:59.307034 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:59.309463 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:59.312343 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:04:59.313660 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.314579 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:59.321188 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:59.331876 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:59.390520 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:04:59.413210 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:59.413302 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:59.416684 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:04:59.424018 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:04:59.425813 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:04:59.436640 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:04:59.438839 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:59.442060 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.532889 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:59.537537 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:59.538197 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:59.539198 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.540710 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.551249 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:59.596383 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:59.601364 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:59.602140 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.612582 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:59.613574 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:59.618113 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.690179 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:04:59.694339 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.701982 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:59.702273 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:04:59.702358 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:59.704531 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.752875 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:04:59.753499 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:04:59.758649 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:04:59.759697 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:04:59.761104 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:04:59.761803 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.762413 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:04:59.778738 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:04:59.793376 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:04:59.800323 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:04:59.828214 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:59.838034 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:04:59.843152 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:04:59.860119 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:04:59.860962 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:04:59.949987 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:04:59.999852 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.003448 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:00.003813 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 14:05:00.020288 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:00.024875 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:05:00.027662 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:00.029557 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.036546 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:00.115576 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:00.118129 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:00.123337 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.126825 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:00.133208 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:00.141745 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:00.150826 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:00.159732 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.160013 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:00.163091 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.171964 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:00.175776 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:00.191684 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:00.195451 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:05:00.206918 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:05:00.274240 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:00.276188 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.279594 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.279609 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:00.290381 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:00.295542 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:00.325124 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:00.337814 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.339493 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:00.343717 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:00.345518 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:00.356523 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.359149 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:00.402972 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:00.424234 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:05:00.446727 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:00.452365 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:00.453152 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.462496 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.465986 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:00.479725 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:00.517341 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:05:00.567903 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:05:00.580611 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.581264 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:00.582717 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.588527 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:00.594987 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:00.619755 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:00.651093 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.655718 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:00.658216 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:00.671382 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.671945 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:00.673847 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:00.706868 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:00.772405 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:00.775019 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.775169 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:00.787709 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.787709 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:00.798825 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:00.868765 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:05:00.913479 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:00.919380 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:00.921705 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:00.925958 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:00.926028 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:00.927385 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:00.928818 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:05:00.935756 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:00.936172 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:00.962814 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:05:00.987509 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:01.036890 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:01.037281 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:05:01.037446 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:01.037791 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.048161 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:01.051632 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:01.060638 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:01.088922 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:01.118714 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:05:01.146019 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.146116 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:01.148744 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:01.155397 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:01.155714 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:01.177171 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.207127 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:01.211423 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.212280 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:01.216223 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:01.216235 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:01.249720 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.328635 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.329416 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:01.334447 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:01.334723 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:01.334684 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:01.352785 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:01.376830 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:01.392035 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:01.395464 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.397128 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:01.398006 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:01.410014 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.417240 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:01.441835 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:01.488133 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240503 14:05:01.488831 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:05:01.491219 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:01.521118 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:01.522038 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:01.522146 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:01.525579 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.525756 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.529815 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:01.534916 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:01.573927 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:05:01.624460 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:05:01.646700 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:01.648587 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:01.654737 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:01.657649 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:01.660182 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.670130 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.763068 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:01.766883 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.770927 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:01.774271 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.776484 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:01.776486 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:01.835453 3039 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:05:01.835609 3039 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:05:01.836155 3039 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:05:01.836673 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:05:01.836819 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:05:01.836892 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:05:01.837035 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:01.837267 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:01.837344 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:01.837491 3039 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:05:01.873340 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:01.874773 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:01.876248 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:01.876698 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:01.883394 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:01.887360 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:01.895556 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.989526 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:01.989781 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:01.991184 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:01.991225 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:01.991627 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:01.995028 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:02.007913 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:02.012311 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:02.019330 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:02.066342 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:05:02.076865 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:05:02.077407 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:02.122587 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:02.123212 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:05:02.130216 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:02.130864 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:02.132107 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.134115 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:02.139505 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.141409 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.153671 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:02.155019 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.175679 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:02.178339 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:02.202322 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:02.255534 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:02.287379 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:02.293543 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:02.293555 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.297360 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.299131 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:02.316879 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:02.323849 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:02.355558 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.359025 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:02.359601 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:02.361037 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.361476 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:02.370168 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:02.381012 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:02.383515 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:02.395288 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:02.462611 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:02.462612 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:02.462965 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:02.464849 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:02.473155 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:02.474112 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.479329 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:02.489902 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:02.496505 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:02.559142 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:05:02.588030 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:05:02.650461 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:05:02.676350 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.676903 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.676936 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:02.681402 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:02.689430 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:02.689819 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:02.774185 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:02.804450 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.807646 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:02.808634 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:02.808768 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:02.821282 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.836247 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.844506 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:02.844885 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:02.851294 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:02.857569 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.861814 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:02.871484 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:02.889182 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:02.893085 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:02.895164 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:02.976866 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:02.985446 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:02.986222 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:02.986326 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.988512 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:02.999377 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:03.047802 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:03.103324 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:03.106245 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.107069 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:03.110431 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:03.114929 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:03.138113 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:03.167497 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:03.192257 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:03.192301 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:03.193499 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.206236 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:03.211964 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:03.225452 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.286921 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:03.291267 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:03.291270 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.294860 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:03.295802 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:03.299562 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:03.315065 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:03.335924 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:05:03.347733 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:05:03.353997 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:03.358889 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.367884 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:03.375455 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:03.382119 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:03.383510 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:03.428692 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:03.432071 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:03.461448 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:03.463486 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:03.464968 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.468257 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:03.472970 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:03.475905 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:05:03.491420 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:03.597259 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:03.659843 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:05:03.671851 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:05:03.688484 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:03.704437 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:03.705828 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:03.707381 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:03.714021 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.721468 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:03.735812 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.813720 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:03.818257 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.818547 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.831959 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:03.833915 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:03.842738 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:03.857335 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:05:03.885552 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:03.887387 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:03.888768 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:03.888912 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.895685 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.907613 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:03.988286 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:03.988837 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:03.989615 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:03.990751 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:03.991421 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.001580 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:04.006055 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:04.056107 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:04.064352 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 14:05:04.064966 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:04.101492 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:04.106791 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:04.109237 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.110684 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.112300 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:04.113649 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:04.150319 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:04.160462 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:05:04.206107 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:04.240187 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:04.242883 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:04.243412 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.245584 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.248742 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:04.261518 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:04.268772 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.279177 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:04.282646 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:04.283032 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.284691 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:04.316109 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:04.343710 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:04.396036 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:04.422648 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:04.424885 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.434216 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:04.439787 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:04.454962 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.455405 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.458281 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:04.458972 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:04.476227 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:04.482089 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:04.492516 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.496491 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:04.569080 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:04.593892 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:04.598124 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:04.785874 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:04.787684 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:05:04.805418 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:04.807740 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:04.808455 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.816426 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:04.819926 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:04.826213 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:04.826704 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:05:04.829283 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:04.829602 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:04.830696 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:04.844192 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:04.846516 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:04.846818 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:04.855748 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:04.856179 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:04.859663 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:04.953655 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:04.974601 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:04.979461 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.984928 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:04.989687 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:04.996083 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:05.003996 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:05.008397 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:05.032467 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:05.035524 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:05.050029 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:05.050066 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.050503 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:05.060094 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:05:05.060564 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:05:05.061793 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:05:05.064607 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:05.087539 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:05:05.156853 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:05.159114 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:05.159373 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:05.161484 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.166584 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:05.168259 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:05.175351 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:05:05.218612 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:05.271374 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:05:05.274153 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:05.275090 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.275363 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:05.275775 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.276773 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:05.285439 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:05.320623 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:05.356926 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:05.364356 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:05.365921 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.368942 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:05.371474 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:05.374080 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.375406 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:05.379359 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:05.441474 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:05.442124 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:05.449997 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.450753 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:05.451161 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:05.469753 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:05:05.476462 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:05.549106 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:05.552081 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.555568 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:05.555712 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.560213 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:05.583709 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:05.744721 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:05.758342 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:05.760298 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.764474 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.764539 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:05.766680 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:05.779080 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:05.809711 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:05.821413 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:05.854892 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:05.879211 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.888481 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:05.894238 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.900561 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:05.912945 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:05.914116 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:05.919405 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:05.919806 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.920100 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:05.935956 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:05.948189 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:05.948488 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:05.951342 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:05.951998 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:05:05.969146 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:05.979684 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:05:06.092063 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:06.095090 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:06.097301 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:06.099221 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.113433 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.117113 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:06.123306 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:06.126374 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.142642 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:06.147503 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:06.149664 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:06.152838 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:06.160261 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:06.221411 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:06.251020 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:06.261968 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:06.265842 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.265892 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:06.266880 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:06.275171 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.285543 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:06.372509 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:06.373589 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:06.377591 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:06.381147 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.382257 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.387979 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:06.400188 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:06.427397 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:06.441442 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:05:06.456611 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:06.484381 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:06.484710 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.487772 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.489190 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:06.492478 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:06.499581 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:06.557901 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 160 similar messages]
I20240503 14:05:06.592533 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.594622 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:06.595208 5046 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:05:06.595391 5046 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:05:06.595760 5046 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:05:06.596975 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:06.597519 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:06.598598 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:06.598660 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:06.598958 5046 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:05:06.600522 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:06.604311 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.610265 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:06.685314 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:06.698937 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:06.703644 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.708122 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:06.712949 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:06.715085 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:06.715934 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:06.725849 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:06.798249 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:05:06.813238 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:06.813699 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:06.814321 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:06.815021 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:06.816534 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.816624 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:06.834841 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:06.883527 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:06.892151 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:06.931156 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:06.936316 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:06.941416 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:06.948781 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.950436 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:06.951413 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.976276 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.981081 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:06.984153 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:06.989019 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:06.996456 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:07.006314 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:07.105423 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:07.113020 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:07.115417 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.118516 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:07.119596 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:07.132154 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:07.140715 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:07.149612 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:07.168154 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:07.171521 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:07.184513 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:07.185736 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.197703 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:07.200327 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:07.268517 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:07.291221 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:07.292359 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.293103 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:07.295187 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:07.296631 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:07.303632 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.345479 6248 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:05:07.345675 6248 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:05:07.346239 6248 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:05:07.348264 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:05:07.348389 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:05:07.348457 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:05:07.348613 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:07.348688 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:07.348762 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:07.348898 6248 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:05:07.364068 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:07.367913 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:07.368189 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:07.396143 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:07.397325 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:07.445019 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:07.446676 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:07.449446 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:07.456409 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:05:07.458920 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.463414 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:07.465391 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:05:07.470999 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:07.480074 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:05:07.496250 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:05:07.528851 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:07.551908 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:07.571705 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:07.571776 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.572546 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:07.578980 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.592180 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:07.594478 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:07.638255 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:07.642204 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:07.643602 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.644125 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:07.645042 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:07.646795 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.657089 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:07.663341 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:07.668552 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:07.760023 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.761390 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:07.762296 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.762342 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:07.762914 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:07.763170 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:07.871838 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.875671 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:07.876499 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:07.892024 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.892203 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:07.892393 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:07.914070 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:07.949014 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:07.951819 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:07.952735 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:07.963866 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:07.963869 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:07.964246 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:07.974364 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:08.070570 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.072283 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:08.073046 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:08.076834 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.091272 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:08.099581 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:08.102140 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:08.124001 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:08.129797 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.132344 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:08.132684 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.138796 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:08.159350 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:08.218775 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:08.236835 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:08.239168 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.240401 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.241092 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:08.243768 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:08.247798 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:08.267833 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:08.431208 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:08.439157 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:08.440991 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:08.455252 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:08.457566 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:08.466322 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.466625 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:08.476523 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:08.476994 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:08.478461 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:08.505095 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:05:08.559161 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:08.576917 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.583896 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.589148 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:08.591493 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:08.596105 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:08.600379 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:08.631552 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:08.636018 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.637907 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.655077 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:08.655740 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:08.656772 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:08.741765 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:08.745604 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:08.749768 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.752282 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.764019 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:08.766364 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:08.788784 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:08.825443 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:08.846449 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:08.855911 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:08.856158 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:08.858875 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:08.865347 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:08.865533 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.871464 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:08.878659 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:05:08.897820 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:08.958365 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:08.979739 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:08.990449 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:08.994532 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:08.999424 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.009439 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:09.011865 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:09.018812 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:09.034224 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:09.034529 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:09.037194 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:09.037570 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.066587 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:09.125361 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:09.175828 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:09.178189 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:09.182441 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.192257 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:09.193295 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 14:05:09.194701 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:09.204418 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:09.206333 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:09.219915 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:09.224016 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.229851 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:09.233901 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.235759 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:09.247655 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:09.306134 5046 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:05:09.306275 5046 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:05:09.306694 5046 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:05:09.307837 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:09.307948 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:09.308010 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:09.308164 5046 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:05:09.310843 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:09.343626 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:09.355672 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:09.463543 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:09.551539 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:09.555929 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.557596 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:09.560685 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.567914 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:09.572563 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:09.574683 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.593650 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:09.593698 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:09.595587 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:09.601584 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:09.607960 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.626160 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:09.719659 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:09.730975 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:09.731889 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:09.736043 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.742149 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:09.746918 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.753023 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.761961 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:09.762387 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:09.786773 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.787206 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:09.801774 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:09.816941 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:09.817107 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:05:09.849015 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:09.899413 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:09.904466 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.905121 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:09.907044 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:09.918882 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:09.920639 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:09.920959 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:09.921576 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:09.921584 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:09.937049 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:09.990424 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:05:09.995321 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:10.000825 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:10.029227 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:10.032235 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:10.032475 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.032583 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:10.032856 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.042284 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:10.064687 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:05:10.092948 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:05:10.097018 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:05:10.104423 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:10.143882 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:10.151093 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:10.158032 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:10.159968 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.163400 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.187752 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:10.192955 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:10.193508 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:10.193701 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:10.197955 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.198932 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:10.206434 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:10.211352 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:10.225522 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:10.308614 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.308799 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:10.311570 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:10.312041 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.314769 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:10.321214 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:10.336045 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:10.365218 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 14:05:10.459152 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:10.526144 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:10.526139 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.526633 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:10.527438 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.528517 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:10.529070 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:10.626662 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:10.630882 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:10.631928 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.632135 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:10.643478 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.657511 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:10.666594 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:10.696774 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:10.701072 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:10.702865 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:10.706681 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:10.708953 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:10.710055 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.723316 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:10.728344 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:10.831112 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:10.831805 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:10.836311 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:10.842495 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.843772 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.852846 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:10.867671 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.867884 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:10.878127 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.878404 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:10.890674 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:10.906786 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:10.994840 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.996507 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:10.996914 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:10.997663 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:10.997745 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:11.000229 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:11.008342 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:11.019501 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:05:11.053094 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:11.094282 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:11.110685 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:11.111522 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.115959 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:11.116627 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:11.127123 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:11.132891 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:11.136346 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:11.138123 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:11.144899 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:11.208917 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:11.210479 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:11.216024 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.216310 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:11.217823 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:11.224493 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:11.230909 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:11.278877 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:11.301343 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:11.320452 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:11.324491 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.327328 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:11.331033 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:11.335391 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:11.335427 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.344405 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:11.401160 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:11.431839 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:11.433086 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.438056 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:11.438920 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:11.452862 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.470146 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:11.524364 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:11.555258 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.555487 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:11.556702 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:11.560055 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:11.560186 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.566095 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:11.573011 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20240503 14:05:11.606513 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:11.664670 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:11.667181 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:11.668022 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:05:11.668634 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:11.677112 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.681938 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:11.694917 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.705998 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:11.733242 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:11.734427 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:11.737555 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.744325 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:11.755950 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.771129 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:11.845803 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.846472 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:11.847761 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:11.853081 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.860754 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:11.861991 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:11.876986 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:11.890826 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:11.929147 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:11.929368 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.929563 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:11.931591 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:11.937510 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:11.952093 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:11.999342 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:05:12.060987 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:12.062568 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.063885 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:12.064332 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.064538 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:12.073737 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:12.083735 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:12.147694 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:12.177459 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:12.178248 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.180346 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:12.180608 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.181277 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:12.187409 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:12.198156 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:12.284334 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:12.287087 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.292218 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:12.293704 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.296790 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:12.312213 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:12.347811 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:12.390486 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:12.390650 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:12.394999 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.401945 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:12.407289 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:12.407729 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:12.409024 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:12.413621 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:05:12.428395 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:05:12.455329 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:05:12.457247 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:12.467936 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:12.507941 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:12.512864 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.514386 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:12.518059 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:12.519677 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.540011 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:12.546991 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:05:12.548287 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:12.560603 7419 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:05:12.560741 7419 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:05:12.561241 7419 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:05:12.562240 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:05:12.562363 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:05:12.562444 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:05:12.563463 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:12.563618 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:12.563688 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:12.563823 7419 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:05:12.587757 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:12.624576 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:12.624593 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:12.624686 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.631764 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:12.633365 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.637580 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:12.641784 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:12.649192 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:12.666679 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:12.679841 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.683611 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:12.688025 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:12.695760 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:12.695827 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:12.710713 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.719108 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:12.822571 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.822861 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:12.824244 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:12.824751 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:12.836266 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.840655 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:12.849400 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:12.857635 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.865373 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:12.875633 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:12.886619 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:12.886828 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:12.892894 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.986971 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:12.989001 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:12.989727 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:12.991297 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:12.991362 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.021245 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:13.153477 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:13.174085 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:13.175650 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.175874 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:13.181130 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:13.182912 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:13.183037 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:13.244081 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:13.294337 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.294775 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:13.299484 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:13.299703 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.301653 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:13.318089 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:13.357635 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:13.358808 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:13.361784 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.364271 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.374922 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:13.377032 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:13.377748 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:13.387077 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:13.417958 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:05:13.483809 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:13.485723 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.487478 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.488370 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:13.497375 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:13.508484 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:13.515182 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:05:13.598024 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:13.604697 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:13.606515 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:13.608448 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:05:13.610029 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:13.611611 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:13.622020 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:13.636718 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.690765 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:13.691301 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:13.694792 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:13.698777 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:13.703411 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.706229 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:13.710175 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:05:13.757311 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:13.789204 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.789587 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:13.790990 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:13.794974 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.815099 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:13.826480 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:13.836261 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:13.849860 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:13.851297 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:05:13.855866 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.860668 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:13.862419 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.866006 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:13.873345 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:13.899879 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:13.906148 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:05:13.958636 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:13.958925 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:05:13.960234 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:13.967387 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:13.968135 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:13.973912 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:13.993912 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:14.083446 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:14.097806 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:14.165454 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.167752 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:14.172006 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:14.172068 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:14.181368 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:14.183559 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:14.188376 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.205489 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:14.226275 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:05:14.227310 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:14.293273 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:05:14.302323 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:14.302394 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:14.304095 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:14.304220 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.313664 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.321046 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:14.346556 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:14.347410 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.350457 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.358115 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:14.366230 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:05:14.373914 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:14.374202 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:14.461220 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:14.466459 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.468113 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:14.468267 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:14.476364 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.484692 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:14.574262 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:14.575320 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.576433 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:14.576884 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:14.579578 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:14.604584 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:14.607443 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:14.616194 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:14.619997 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:14.624598 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:14.669692 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:14.687822 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:14.692809 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:14.694808 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:14.695588 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:05:14.699620 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.703145 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.713626 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:14.716885 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:05:14.755966 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:14.764242 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:14.765954 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.766249 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:14.766494 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:14.777024 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:14.800529 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.864401 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:14.873448 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.876520 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.877185 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:14.884347 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:05:14.885198 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:14.903215 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:14.934816 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:14.939945 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:14.951567 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:14.954842 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:14.961279 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:14.970248 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:14.983366 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:15.010159 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:05:15.038818 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:15.107089 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:15.158685 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:15.190667 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:15.218843 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:15.219866 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:15.228446 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:15.235476 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.236779 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:15.238343 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:05:15.242347 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:05:15.248999 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.261093 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:15.313628 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:15.314216 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:15.319151 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.322772 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:15.328322 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:15.330358 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.401130 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:15.401726 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:15.407876 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:15.416631 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:15.416728 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.430677 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.480033 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:15.484077 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.485764 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:15.489481 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:15.491767 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.500605 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:15.503401 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 14:05:15.593139 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:15.598925 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.599622 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:15.600488 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:15.607609 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:15.622125 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:15.645321 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:15.719303 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:15.722450 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.730731 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:15.745198 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:15.746244 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.759701 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:15.785256 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:15.785341 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:15.790297 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.797192 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:15.803483 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:15.804709 5046 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:05:15.804843 5046 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:05:15.805294 5046 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:05:15.806732 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:15.807171 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:15.808252 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:15.808327 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:15.809664 5046 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:05:15.811147 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:15.824141 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:15.900741 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:15.904610 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.904672 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:15.906790 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:15.907987 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.928524 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:15.966900 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:15.972210 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:15.973726 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:15.974206 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.989771 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:15.995138 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:15.998538 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:16.038479 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:16.101068 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:05:16.115350 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:16.160019 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:05:16.174019 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:16.177779 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:16.236970 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:16.238670 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.238687 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.258754 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:16.261895 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:16.263723 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:16.272457 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:05:16.272887 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:16.283697 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:16.328598 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:16.329401 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.330226 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:16.336679 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.337924 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:16.347220 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:16.366674 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:16.399159 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:16.413003 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.416643 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:16.427281 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:16.431663 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:16.431761 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.453714 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:16.474809 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:16.512475 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:16.517189 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:16.521062 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:16.529950 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.531770 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.536176 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:16.542102 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:16.573556 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240503 14:05:16.574347 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:16.589159 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:16.591384 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.592248 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:16.593545 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:16.603672 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.604797 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:16.698895 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.700793 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:16.703801 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.706784 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:16.710621 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:16.716274 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:16.788103 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:16.815829 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:16.821533 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:16.823184 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:16.824288 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.827629 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:16.832633 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:16.835608 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.894750 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:16.895051 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:16.895295 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:16.896515 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:16.907770 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:16.916349 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:16.931311 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:05:16.932500 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:16.935619 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:16.957070 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:17.008976 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:17.011919 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:17.012012 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.026098 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.032557 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:17.035148 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:17.138980 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:17.139935 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.141909 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:17.143872 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:17.151295 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.151518 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:17.171034 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:17.210476 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:05:17.227785 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:17.262120 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.266798 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:17.267083 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:17.269315 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:17.270076 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:17.290401 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:17.299273 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:17.299988 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:17.339468 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:17.352602 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.354729 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.358028 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:17.366614 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:17.373420 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:17.383399 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:17.386776 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:17.432862 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:17.438614 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:17.440966 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:17.448801 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.452579 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.461266 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:17.468374 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:05:17.514245 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:17.545032 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:17.545758 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.546697 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:17.553403 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.565701 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:17.582708 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:17.603443 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:05:17.610188 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:17.610422 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:17.614574 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.616650 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:17.619633 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:17.627846 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.632375 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:17.716672 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:17.724398 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:17.726668 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:17.729025 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.729688 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:17.732031 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:17.747726 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:17.756932 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:05:17.765952 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:17.773730 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:17.780040 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:17.872138 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:17.877148 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:17.880440 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:17.882339 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:17.883396 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.891338 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:17.975450 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:17.990597 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:17.991348 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:17.992240 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:17.998674 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:17.999404 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.008107 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:18.086266 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:18.089351 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.089527 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:18.093041 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:18.093083 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.111722 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:18.197801 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:18.200865 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:18.202621 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:18.210691 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:18.211416 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.214754 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:18.236737 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:18.310137 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:18.342489 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.342738 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:18.344074 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:18.346621 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:18.347273 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:18.352582 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:18.360932 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:18.363166 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:18.368201 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:18.378196 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:18.387048 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.387470 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:18.393152 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.508980 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.509258 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:18.512887 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:18.514819 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.515254 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:18.541386 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:18.544222 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:18.560658 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:18.562170 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:18.565145 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.567646 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:18.578688 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:18.580288 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:18.600713 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:18.612537 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:18.674726 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:18.679158 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.679451 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:18.682089 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.684366 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:18.686970 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:18.695801 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:18.725473 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:18.758073 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:05:18.764449 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:05:18.802995 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:05:18.819442 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:18.835105 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:18.845849 5046 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:05:18.845984 5046 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:05:18.846405 5046 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:05:18.847482 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:18.847566 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:18.847625 12103 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:18.847746 5046 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:05:18.886530 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:18.887760 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:18.887930 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:18.893299 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:18.893528 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:18.908830 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:18.919451 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:18.921397 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:18.959311 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:05:18.987167 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:18.999207 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:19.005000 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:19.011040 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:19.012775 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.025022 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:19.047523 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.063498 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:19.075865 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.076565 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:19.082546 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.090016 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:19.090328 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:19.177392 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:19.177965 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.178941 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:19.180739 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.182696 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:19.191319 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:19.206110 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:19.252714 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 14:05:19.298432 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:19.300117 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:19.302820 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.307386 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:19.310081 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:19.314792 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:19.315603 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:19.406921 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:19.409066 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:19.409598 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:19.415962 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:05:19.421739 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:19.426436 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:19.430018 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.430492 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:19.483515 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:19.487448 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.487542 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:19.489385 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:19.492097 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:19.495436 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:19.513128 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:19.589380 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.592072 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:19.595538 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:19.599056 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.602574 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:19.607100 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:19.622936 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:19.651855 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:19.652392 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:19.654719 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:19.662684 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:19.666443 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:19.667941 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.678941 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:19.721148 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:05:19.722141 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:19.757030 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:19.823597 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:19.847868 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:19.918534 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:19.919119 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.924055 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:19.924579 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:19.932003 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:19.936857 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:19.940811 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:19.991994 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:05:19.998957 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:20.008525 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:20.011961 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:20.018262 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:20.019479 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:20.021011 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.022651 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:20.023223 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.101787 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:20.104418 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:20.112854 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.114466 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.114953 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:20.124408 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:20.149107 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:20.150596 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:20.175503 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:20.188099 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:20.188620 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:20.191105 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.191475 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.197523 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:20.198796 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:20.300818 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.302081 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:20.303118 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:20.311414 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.320111 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:20.326356 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:20.344352 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:20.361126 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:20.378039 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:20.430758 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:20.441129 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:20.443729 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:20.458082 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:20.458259 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.468456 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:20.468987 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:20.470671 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:20.475981 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.504536 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:20.505271 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.506709 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.527446 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:20.615909 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:20.619495 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 14:05:20.626595 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:20.627429 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:20.629421 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.631336 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:20.633860 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.642594 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.652727 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:20.660836 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:20.671651 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:20.675074 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:20.694571 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:20.712229 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:20.747761 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:20.917279 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:20.934576 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:20.935705 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:05:21.001577 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:21.012749 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:21.013826 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.014101 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.022606 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:21.038780 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:21.052876 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:21.053992 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:21.065169 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.068621 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:21.070670 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.072638 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:21.124886 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:21.143471 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:21.163419 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:05:21.175981 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:21.182271 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:21.190421 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:21.190850 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:21.201131 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.217859 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:21.232067 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:05:21.237370 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:21.239329 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.243345 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:21.247637 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:21.276154 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:21.278074 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:21.285997 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:21.304538 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:21.319516 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:21.323946 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:21.347189 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:21.356473 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.356514 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:21.357295 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:21.358515 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:21.359638 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:21.361920 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:21.369030 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:05:21.370121 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:21.468125 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:21.468912 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.472272 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:21.473029 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:21.484239 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:21.485158 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:21.495570 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:21.594182 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:21.596355 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:21.599869 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 159 similar messages]
I20240503 14:05:21.603669 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:21.606802 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.619459 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:21.621984 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:21.622786 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:21.629931 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:21.640486 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.653970 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:21.655869 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.671955 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:21.691176 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:21.706459 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:21.766932 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:21.767812 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:21.769421 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.769565 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.782856 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:21.789644 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:21.816606 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:21.870862 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:21.888657 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:05:21.963586 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:21.965211 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:21.967082 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.979781 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:21.980222 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:21.981292 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:22.032447 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:22.093324 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:22.095124 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.095270 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:22.099138 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:22.111685 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:22.118422 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:22.119999 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:22.131410 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:22.156739 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:22.161260 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:22.161669 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.162181 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:22.167690 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:22.177860 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:22.179767 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:05:22.239498 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:22.284415 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.287880 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:22.293988 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.295483 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:22.296993 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:22.322901 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:22.327462 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:22.336709 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:22.340572 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:05:22.346025 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.349484 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:22.352209 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:22.363098 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:05:22.368355 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:22.374650 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:22.386462 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:22.407447 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:22.466059 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:22.466154 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:22.470104 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.471302 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:22.471747 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:22.478447 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:22.478868 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:22.570763 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:22.578120 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:05:22.582131 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:22.586126 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:22.587534 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.589792 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.590879 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:22.593421 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:22.598801 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:22.611550 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:22.678841 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:22.680092 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:22.682858 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:22.683420 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.688077 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.693099 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:22.796576 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:22.796893 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.799717 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:22.801501 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:22.801784 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.804461 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:22.882234 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:22.888861 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:05:22.903457 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:22.906810 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:22.907428 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.908352 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:22.912730 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:22.931649 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:22.939435 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:05:22.940724 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:23.017657 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:23.022161 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.026324 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:23.037377 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.049818 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:23.052485 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:23.110306 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:23.142547 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:23.143281 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:23.148986 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:23.149235 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:23.177234 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.177234 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.200567 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.203871 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.204599 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:23.209662 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:23.210945 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:23.214859 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:23.281109 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:05:23.316179 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:23.325563 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:23.329265 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:23.330163 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:23.333628 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.341775 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.358140 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:23.375388 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:23.379572 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:23.382256 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.408774 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.409164 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:23.412217 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:23.420572 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:05:23.474931 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:05:23.487695 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:23.517458 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.520537 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:23.523376 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.524089 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:23.535163 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:23.536237 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:23.671991 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:23.674346 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:23.678508 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:23.678776 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:05:23.679908 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.680042 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:23.680536 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:23.692307 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:23.703938 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:23.802673 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:23.804776 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:23.805464 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:23.808816 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:23.808825 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:23.816895 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.838394 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:23.882292 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:23.883190 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.883513 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:23.889446 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:23.889899 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:23.895457 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:23.898965 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:23.953510 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:23.979727 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:24.008991 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:24.009132 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.010169 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:24.010424 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:24.015071 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:24.015437 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:24.071698 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:24.122025 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:05:24.144014 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:24.145380 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.147323 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.150957 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:24.154605 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:24.167459 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:24.168880 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:24.183946 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:24.188732 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:24.189033 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.200101 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.204015 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:24.206260 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:24.252812 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240503 14:05:24.253505 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:05:24.259155 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:24.332211 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:24.337567 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:24.343129 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:24.345320 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:24.348587 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.352202 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:24.357914 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.358526 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:24.373534 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:24.377858 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:05:24.383602 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:24.392997 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.393118 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:24.397171 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:24.493163 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:24.504914 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:24.507931 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:24.511690 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:24.515848 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.517007 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.528246 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:24.652729 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:24.665112 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:05:24.693228 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:24.694517 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:24.695283 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.698768 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.708164 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:24.720049 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:24.758002 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:24.761621 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:24.803112 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:24.817868 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:24.817962 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:24.819406 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:05:24.820502 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:24.821319 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.826817 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.831416 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:24.847613 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:24.848991 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:05:24.850522 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:05:24.869948 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:24.877647 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:24.878402 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.883844 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:24.891436 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:24.893471 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:24.904162 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:24.922677 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:24.995605 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:24.996302 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.008826 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.010852 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:25.012864 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:25.020694 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:25.025802 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:25.087359 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:25.102766 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:25.114019 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.115629 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.117503 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:25.122479 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:25.126380 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:25.134619 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:25.228076 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:25.234227 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.234903 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:25.242178 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:25.266904 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:25.286891 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:25.290279 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:25.290856 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:25.292769 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.295775 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:25.298858 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.299624 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:25.304780 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:25.322928 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:25.323791 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.437875 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:25.447523 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:25.452726 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:25.457867 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.458230 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:25.459899 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:25.476372 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:25.479022 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:05:25.491776 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.508491 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:25.514456 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:25.515122 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:25.517661 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:25.519534 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.677084 14238 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:05:25.677219 14238 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:05:25.677850 14238 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
W20240503 14:05:25.678269 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 14:05:25.678622 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:05:25.678753 12036 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:05:25.678828 12036 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:05:25.679093 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:25.679212 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:25.679314 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:25.679481 14238 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:05:25.735431 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:25.764142 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:05:25.765372 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:25.814214 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:25.817971 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:25.821336 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:25.823047 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:25.828575 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.851089 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.856514 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:25.858595 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:25.859859 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:25.859922 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:25.882354 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:25.891932 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:25.934633 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:05:25.959184 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:05:25.992563 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:26.005842 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:26.006027 12410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.006129 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.007936 13754 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:26.008082 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:26.021076 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:26.032079 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:26.038038 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:26.040683 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.044994 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:26.073578 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.082479 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:26.092694 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:05:26.146046 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:26.167922 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:05:26.196143 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:26.196836 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:26.199910 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:26.209172 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:26.209723 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.224805 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:26.280387 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:26.314587 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:26.319531 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:26.320168 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:26.324012 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.331214 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:26.331437 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:26.334625 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:05:26.335669 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:26.431707 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:26.432608 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:26.441740 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.448693 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:26.450970 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:26.465487 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:26.477902 12235 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:26.509569 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:26.514209 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.525527 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:26.525790 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:26.528406 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:26.541668 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:26.574103 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:26.597529 12235 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:05:26.602711 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:26.603279 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:26.603888 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.604169 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:26.617319 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.627063 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:26.670053 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 162 similar messages]
W20240503 14:05:26.713519 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:05:26.744156 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:26.761950 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:26.833231 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:26.834198 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:26.836640 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:26.836652 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.838733 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:26.848302 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:26.862849 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:26.893760 11973 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:05:26.946352 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:26.946527 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:26.956554 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:26.959407 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:26.960304 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:26.963423 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:26.996433 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:27.024984 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:27.028353 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:27.031926 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:27.037577 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:05:27.039044 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.041124 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:27.053066 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:27.073410 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:05:27.091647 12366 consensus_peers.cc:485] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:27.130098 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:27.135071 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.137630 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.140796 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:27.140909 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:27.141804 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:27.165381 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:27.194836 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:05:27.201403 12366 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:05:27.203397 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:05:27.208545 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:27.211472 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:27.214438 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.217515 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:27.217705 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.243340 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:27.254434 12235 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:05:27.288556 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:27.331168 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:27.333231 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.334796 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:27.338260 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:27.347720 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.360392 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:27.421166 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:05:27.473157 12235 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:05:27.473837 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:27.474191 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:27.474239 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:27.477638 11973 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:05:27.490962 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:27.493438 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:27.495416 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:27.495837 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:27.497900 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.499349 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:27.512607 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:05:27.527180 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:27.531955 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:27.539461 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:27.557036 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:27.566354 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:27.631038 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:05:27.644057 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.644115 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:27.644322 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:27.647843 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:27.649593 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:27.667407 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.816829 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:27.818560 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.820493 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.821070 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:27.834666 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:27.841678 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:27.843080 12235 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:05:27.879812 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:27.926864 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:27.931684 12104 consensus_peers.cc:485] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 14:05:27.942029 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.943742 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:27.944340 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:27.945317 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:27.945922 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:27.962080 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:28.016036 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:28.030675 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.033332 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.035274 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:28.035446 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:28.049669 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:28.103858 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:28.107014 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:28.108100 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:28.110908 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:28.111793 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.129086 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:28.185377 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:28.224825 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:28.233956 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.235913 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:28.236160 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.237977 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:28.240084 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:28.259994 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 14:05:28.263607 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 14:05:28.270502 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.280663 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:28.283407 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:28.283488 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:28.286902 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.299407 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:28.374758 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 14:05:28.383587 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:28.385998 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 14:05:28.388000 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:28.388609 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:28.394961 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.395548 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.419301 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:28.489169 11973 consensus_peers.cc:462] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:28.489169 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:28.580718 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:28.581487 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.581507 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.587460 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:28.588225 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:28.591719 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:28.691289 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:28.695653 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:28.703482 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:28.704749 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:28.707501 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:28.718475 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.741297 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:28.751601 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:28.757725 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:28.762580 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.762689 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:28.762969 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.763118 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:28.836033 12234 consensus_peers.cc:485] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:05:28.870010 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.870291 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:28.870507 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:28.872154 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:28.880479 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:28.887991 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.912608 14896 raft_consensus.cc:998] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.037s)
I20240503 14:05:28.913059 14896 consensus_queue.cc:237] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:05:28.914534 13774 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:05:28.914566 12428 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:05:28.914870 13906 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:05:28.914964 15371 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:05:28.915123 14896 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:05:28.915575 14896 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:05:28.917696 14896 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:28.918049 13906 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:28.919736 12428 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:28.920957 11894 catalog_manager.cc:5484] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 9 to 10, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:28.921538 13774 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
W20240503 14:05:28.928982 11882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:28.930231 11882 catalog_manager.cc:4573] TS e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet faa0a6ca00d442eaaf9fca7fc1259ec7: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:28.980870 12633 raft_consensus.cc:998] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.133s)
I20240503 14:05:28.981240 12633 consensus_queue.cc:237] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:05:28.982574 12036 raft_consensus.cc:1261] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:28.982712 13774 raft_consensus.cc:1261] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:28.985989 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:28.986182 12633 consensus_queue.cc:1035] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:28.986507 12633 consensus_queue.cc:1035] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:28.988677 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:28.988759 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:28.990187 15598 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:28.990711 13774 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:28.993461 11892 catalog_manager.cc:5484] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:28.997794 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:28.999720 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:29.004386 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 14:05:29.005946 12036 raft_consensus.cc:2932] T f4f9cb87c70e43dbb0d850c275ebb8a5 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.015528 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.073748 14821 raft_consensus.cc:998] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.085s)
I20240503 14:05:29.074139 14821 consensus_queue.cc:237] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:05:29.076246 12166 raft_consensus.cc:1261] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:29.077207 15068 consensus_queue.cc:1035] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:05:29.079389 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.080768 15641 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.082491 12166 raft_consensus.cc:2932] T b031f09b97664d1a94c14f0661181899 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.083665 11888 catalog_manager.cc:5484] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
W20240503 14:05:29.093665 11972 consensus_peers.cc:462] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:29.106640 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:29.106634 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:29.108206 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.112044 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.126109 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:29.127732 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:05:29.133980 12235 consensus_peers.cc:462] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:29.164861 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:29.168013 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:29.169066 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:29.169232 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.173668 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:29.176290 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.195372 15641 raft_consensus.cc:998] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.083s)
I20240503 14:05:29.195737 15641 consensus_queue.cc:237] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:05:29.201856 12166 raft_consensus.cc:1261] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:29.201936 13906 raft_consensus.cc:1261] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:29.202267 15653 consensus_queue.cc:1035] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:29.202459 14821 consensus_queue.cc:1035] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:29.204360 14821 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.207028 11889 catalog_manager.cc:5484] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.211881 12037 consensus_queue.cc:237] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:05:29.212018 12166 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.213133 13775 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:05:29.213186 13904 raft_consensus.cc:2932] T adf2c44328e244779955e712b99f8331 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.213455 13906 raft_consensus.cc:1261] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:05:29.215651 15371 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:05:29.216133 15545 consensus_queue.cc:1035] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:05:29.217682 15371 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.217814 13775 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.217908 13906 raft_consensus.cc:2932] T faa0a6ca00d442eaaf9fca7fc1259ec7 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.220314 11889 catalog_manager.cc:5484] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 10 to 11, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.228109 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet faa0a6ca00d442eaaf9fca7fc1259ec7 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:42886
I20240503 14:05:29.228458 15737 tablet_replica.cc:330] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:05:29.228639 15737 raft_consensus.cc:2229] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:29.228916 15737 raft_consensus.cc:2258] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:29.230798 15737 ts_tablet_manager.cc:1875] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:29.233295 15641 raft_consensus.cc:998] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.121s)
I20240503 14:05:29.233592 15641 consensus_queue.cc:237] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
W20240503 14:05:29.236653 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.237151 15737 ts_tablet_manager.cc:1888] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 14:05:29.237234 15737 log.cc:1196] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/faa0a6ca00d442eaaf9fca7fc1259ec7
I20240503 14:05:29.237732 13775 raft_consensus.cc:1261] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:05:29.237915 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet faa0a6ca00d442eaaf9fca7fc1259ec7 (table rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]) successfully deleted
I20240503 14:05:29.238603 15653 consensus_queue.cc:1035] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:05:29.240434 15653 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.240841 13775 raft_consensus.cc:2932] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.243577 11891 catalog_manager.cc:5484] T 79b21b1e5292488aad442060b417f3c1 P 66190085d1c04e068d5c0dec9c4bf07d reported cstate change: config changed from index 8 to 9, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.244513 14666 raft_consensus.cc:998] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.019s)
I20240503 14:05:29.244750 14666 consensus_queue.cc:237] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:05:29.246205 13906 raft_consensus.cc:1261] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:05:29.246588 15756 consensus_queue.cc:1035] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:05:29.248651 14666 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.250841 13906 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.250972 11889 catalog_manager.cc:5484] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.253705 12428 consensus_queue.cc:237] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
W20240503 14:05:29.255041 12366 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): Couldn't send request to peer 8b17d706d2344dbaae13cda9b0733143. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70ffc5888a5945d6a8e166420e6f1bd9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.255298 13906 raft_consensus.cc:1261] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:05:29.255843 15756 consensus_queue.cc:1035] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 14:05:29.256438 12234 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:05:29.257540 14666 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:05:29.257974 13906 raft_consensus.cc:2932] T 70ffc5888a5945d6a8e166420e6f1bd9 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:05:29.258069 15545 raft_consensus.cc:998] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.293s)
I20240503 14:05:29.258394 15545 consensus_queue.cc:237] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:05:29.259469 11879 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 70ffc5888a5945d6a8e166420e6f1bd9 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240503 14:05:29.259773 11973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240503 14:05:29.260129 11889 catalog_manager.cc:5484] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 8 to 9, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:05:29.260253 13904 raft_consensus.cc:1261] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:29.260426 12166 raft_consensus.cc:1261] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:29.260612 15544 consensus_queue.cc:1035] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:29.260756 15371 consensus_queue.cc:1035] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:29.263545 15371 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.263967 12166 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.264447 13904 raft_consensus.cc:2932] T 0e442bb664f44887aa8afdb62e1d97fc P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.266268 11891 catalog_manager.cc:5484] T 0e442bb664f44887aa8afdb62e1d97fc P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.319644 15068 raft_consensus.cc:998] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.323s)
I20240503 14:05:29.320065 15068 consensus_queue.cc:237] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:05:29.322216 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:29.322794 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.323269 12037 raft_consensus.cc:1261] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:29.323760 15653 consensus_queue.cc:1035] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:29.325143 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.325839 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.329162 15371 raft_consensus.cc:998] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.401s)
I20240503 14:05:29.329500 15371 consensus_queue.cc:237] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:05:29.330514 13904 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:05:29.331279 15545 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:05:29.336815 15653 raft_consensus.cc:998] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.278s)
I20240503 14:05:29.337136 15653 consensus_queue.cc:237] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:05:29.339510 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:29.341118 13906 raft_consensus.cc:1261] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240503 14:05:29.342387 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.342588 15068 consensus_queue.cc:1035] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:05:29.347656 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:29.358592 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:29.359176 15068 raft_consensus.cc:2932] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.363270 12037 raft_consensus.cc:2932] T d1e9155230c74e92b6ee441de713a419 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.363384 11895 catalog_manager.cc:5484] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.368288 15545 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.369728 15068 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.372066 13904 raft_consensus.cc:2932] T a0106f7a40cb467eb4738890ac4e22d4 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.372725 11895 catalog_manager.cc:5484] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.373137 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:29.373589 13906 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.374648 11891 catalog_manager.cc:5484] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 12 to 13, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.376312 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:29.385897 12037 consensus_queue.cc:237] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:05:29.389086 13906 raft_consensus.cc:1261] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:05:29.389482 14896 consensus_queue.cc:1035] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:05:29.390969 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.391620 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:29.394042 14896 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:05:29.395830 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a4b4e16598b740cb9d690c2442a3e994 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:05:29.396050 13906 raft_consensus.cc:2932] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:05:29.399843 11893 catalog_manager.cc:5484] T a4b4e16598b740cb9d690c2442a3e994 P b67f337e10994234baedbf0b19c13513 reported cstate change: config changed from index 8 to 9, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:05:29.400053 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:29.403384 11973 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399): Couldn't send request to peer ca5386f84994496fa5376624391728ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4b4e16598b740cb9d690c2442a3e994. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.403628 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:29.440285 15760 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:05:29.440409 15760 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:05:29.440897 15760 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
W20240503 14:05:29.441680 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:29.441668 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:05:29.441773 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:05:29.441838 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:05:29.442293 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:29.442349 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:29.442549 15760 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:05:29.450125 15545 raft_consensus.cc:998] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.133s)
I20240503 14:05:29.450521 15545 consensus_queue.cc:237] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:05:29.451838 12166 raft_consensus.cc:1261] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 623e8dcb58544e558207d8e652dfed5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:29.452221 14896 consensus_queue.cc:1035] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:05:29.452329 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.454013 14896 raft_consensus.cc:2932] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.455909 12166 raft_consensus.cc:2932] T d837845c6f644bca83315c9f7c45246d P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.457288 11895 catalog_manager.cc:5484] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f reported cstate change: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "623e8dcb58544e558207d8e652dfed5f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.515041 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.517449 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:29.517832 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:29.517854 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:29.518344 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:29.525933 15760 raft_consensus.cc:998] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.231s)
I20240503 14:05:29.526291 15760 consensus_queue.cc:237] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } }
I20240503 14:05:29.528712 12166 raft_consensus.cc:1261] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:05:29.529145 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.529568 14666 consensus_queue.cc:1035] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:29.538187 14666 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.540129 12166 raft_consensus.cc:2932] T 11a8c270532947ba9b505aace44b56f6 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.542881 11894 catalog_manager.cc:5484] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } }
I20240503 14:05:29.550607 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.590281 15881 raft_consensus.cc:998] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.359s)
I20240503 14:05:29.590651 15881 consensus_queue.cc:237] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:05:29.591897 13775 raft_consensus.cc:1261] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:29.592274 15756 consensus_queue.cc:1035] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:29.592636 12037 raft_consensus.cc:1261] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Refusing update from remote peer 84c63c6f551949f6a1d7c07760824361: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:29.593009 15756 consensus_queue.cc:1035] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:29.593880 15760 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.594310 13775 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.594936 12037 raft_consensus.cc:2932] T 4224ca4d93ce4784b52a7550468ce1b1 P 623e8dcb58544e558207d8e652dfed5f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.597153 11889 catalog_manager.cc:5484] T 4224ca4d93ce4784b52a7550468ce1b1 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 6 to 7, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "84c63c6f551949f6a1d7c07760824361" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.605137 15598 raft_consensus.cc:998] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143: Attempting to remove follower 1b4a61e8a11440e7a9f4b300d2442f75 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (300.128s)
I20240503 14:05:29.605437 15598 consensus_queue.cc:237] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:05:29.606463 13906 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 3 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 14:05:29.606942 15600 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:05:29.609148 15598 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.609917 13906 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.612334 11894 catalog_manager.cc:5484] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 reported cstate change: config changed from index 9 to 10, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New cstate: current_term: 3 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:29.619997 12166 consensus_queue.cc:237] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:05:29.621201 13906 raft_consensus.cc:1261] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 3 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
W20240503 14:05:29.621481 12102 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133:37005): Couldn't send request to peer 66190085d1c04e068d5c0dec9c4bf07d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c4f3bc64e7145e2b692beab730d1496. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.621624 15598 consensus_queue.cc:1035] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:05:29.622999 15600 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.623412 13906 raft_consensus.cc:2932] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.625217 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6c4f3bc64e7145e2b692beab730d1496 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:05:29.625903 11895 catalog_manager.cc:5484] T 6c4f3bc64e7145e2b692beab730d1496 P b67f337e10994234baedbf0b19c13513 reported cstate change: config changed from index 10 to 11, VOTER 66190085d1c04e068d5c0dec9c4bf07d (127.0.99.133) added. New cstate: current_term: 3 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.631354 14821 raft_consensus.cc:998] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee: Attempting to remove follower e85b0283a0fd4e3ba81b4170f3cdacb5 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.484s)
I20240503 14:05:29.631734 14821 consensus_queue.cc:237] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:05:29.633024 13775 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:05:29.633355 12428 raft_consensus.cc:1261] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:05:29.633476 15068 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:05:29.634693 15068 consensus_queue.cc:1035] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 14:05:29.636835 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.637485 15653 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:29.640475 13775 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:29.640820 11895 catalog_manager.cc:5484] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 9 to 10, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:29.642232 15641 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (300.169s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:29.644177 12428 raft_consensus.cc:2932] T a6a89230883347d4a64d96d321247012 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:29.644649 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.648355 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:29.648542 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:29.658567 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:29.660907 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.663390 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:29.667693 15920 raft_consensus.cc:998] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee: Attempting to remove follower 1b4a61e8a11440e7a9f4b300d2442f75 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (300.163s)
I20240503 14:05:29.667979 15920 consensus_queue.cc:237] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:05:29.668920 13775 raft_consensus.cc:1261] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:05:29.669275 14821 consensus_queue.cc:1035] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:05:29.687899 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.219s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:29.689579 15068 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.691439 13775 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.692112 11889 catalog_manager.cc:5484] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 7 to 8, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.700241 12299 consensus_queue.cc:237] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:05:29.701257 13775 raft_consensus.cc:1261] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 14:05:29.701599 12235 consensus_peers.cc:485] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee -> Peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263): Couldn't send request to peer 8b17d706d2344dbaae13cda9b0733143. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e92d8f55591346158b9583feb958a8ca. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.702234 14821 consensus_queue.cc:1035] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:05:29.705106 15641 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:05:29.705539 13775 raft_consensus.cc:2932] T e92d8f55591346158b9583feb958a8ca P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:05:29.706530 15956 ts_tablet_manager.cc:902] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143: Initiating tablet copy from peer 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927)
I20240503 14:05:29.706701 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e92d8f55591346158b9583feb958a8ca with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:05:29.707700 11893 catalog_manager.cc:5484] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 8 to 9, VOTER 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } }
I20240503 14:05:29.708918 15956 tablet_copy_client.cc:308] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:33927
I20240503 14:05:29.709530 12450 tablet_copy_service.cc:140] P 84c63c6f551949f6a1d7c07760824361: Received BeginTabletCopySession request for tablet 70ffc5888a5945d6a8e166420e6f1bd9 from peer 8b17d706d2344dbaae13cda9b0733143 ({username='slave'} at 127.0.99.130:47133)
I20240503 14:05:29.709632 12450 tablet_copy_service.cc:161] P 84c63c6f551949f6a1d7c07760824361: Beginning new tablet copy session on tablet 70ffc5888a5945d6a8e166420e6f1bd9 from peer 8b17d706d2344dbaae13cda9b0733143 at {username='slave'} at 127.0.99.130:47133: session id = 8b17d706d2344dbaae13cda9b0733143-70ffc5888a5945d6a8e166420e6f1bd9
W20240503 14:05:29.710465 12103 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 14:05:29.711102 12450 tablet_copy_source_session.cc:205] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:05:29.713550 15956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ffc5888a5945d6a8e166420e6f1bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:29.716542 15956 tablet_copy_client.cc:783] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Starting download of 5 data blocks...
I20240503 14:05:29.721516 15956 tablet_copy_client.cc:647] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:29.722755 15956 tablet_copy_client.cc:523] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:29.723948 15956 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:05:29.736452 15956 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:29.736665 15956 tablet_bootstrap.cc:492] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143: Bootstrap complete.
I20240503 14:05:29.736814 15956 ts_tablet_manager.cc:1367] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20240503 14:05:29.737111 15956 raft_consensus.cc:348] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:05:29.737205 15956 raft_consensus.cc:729] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:05:29.737344 15956 consensus_queue.cc:260] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:05:29.737586 15956 ts_tablet_manager.cc:1398] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 14:05:29.738044 12450 tablet_copy_service.cc:342] P 84c63c6f551949f6a1d7c07760824361: Request end of tablet copy session 8b17d706d2344dbaae13cda9b0733143-70ffc5888a5945d6a8e166420e6f1bd9 received from {username='slave'} at 127.0.99.130:47133
I20240503 14:05:29.738121 12450 tablet_copy_service.cc:433] P 84c63c6f551949f6a1d7c07760824361: ending tablet copy session 8b17d706d2344dbaae13cda9b0733143-70ffc5888a5945d6a8e166420e6f1bd9 on tablet 70ffc5888a5945d6a8e166420e6f1bd9 with peer 8b17d706d2344dbaae13cda9b0733143
I20240503 14:05:29.747694 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(70ffc5888a5945d6a8e166420e6f1bd9): 4838 bytes on disk
I20240503 14:05:29.748068 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(70ffc5888a5945d6a8e166420e6f1bd9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240503 14:05:29.774165 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:29.775480 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:29.778841 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.780221 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:29.790936 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.818845 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:29.824927 16033 ts_tablet_manager.cc:902] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee: Initiating tablet copy from peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:05:29.826737 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:29.834040 16033 tablet_copy_client.cc:308] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:36489
I20240503 14:05:29.835774 12057 tablet_copy_service.cc:140] P 623e8dcb58544e558207d8e652dfed5f: Received BeginTabletCopySession request for tablet a4b4e16598b740cb9d690c2442a3e994 from peer ca5386f84994496fa5376624391728ee ({username='slave'} at 127.0.99.131:56133)
I20240503 14:05:29.835865 12057 tablet_copy_service.cc:161] P 623e8dcb58544e558207d8e652dfed5f: Beginning new tablet copy session on tablet a4b4e16598b740cb9d690c2442a3e994 from peer ca5386f84994496fa5376624391728ee at {username='slave'} at 127.0.99.131:56133: session id = ca5386f84994496fa5376624391728ee-a4b4e16598b740cb9d690c2442a3e994
I20240503 14:05:29.836315 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:29.837131 12057 tablet_copy_source_session.cc:205] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:05:29.838181 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:29.838342 16033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b4e16598b740cb9d690c2442a3e994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:29.841260 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:29.842952 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.855252 14821 raft_consensus.cc:998] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee: Attempting to remove follower 1b4a61e8a11440e7a9f4b300d2442f75 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (300.480s)
I20240503 14:05:29.855554 14821 consensus_queue.cc:237] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:05:29.856843 12430 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:05:29.857575 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.857863 15920 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:29.872743 16033 tablet_copy_client.cc:783] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee: tablet copy: Starting download of 5 data blocks...
I20240503 14:05:29.874008 15641 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:05:29.875401 12430 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:05:29.876472 11886 catalog_manager.cc:5484] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 5 to 7, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New cstate: current_term: 2 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:05:29.880084 12299 consensus_queue.cc:237] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
W20240503 14:05:29.881232 12235 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489): Couldn't send request to peer 623e8dcb58544e558207d8e652dfed5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8933689df2c45b59a9cd16024ddb7d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:29.881651 12430 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:05:29.882193 15641 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:05:29.883626 15920 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:05:29.884760 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c8933689df2c45b59a9cd16024ddb7d9 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:05:29.885010 12430 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:05:29.888448 11887 catalog_manager.cc:5484] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 7 to 8, VOTER 623e8dcb58544e558207d8e652dfed5f (127.0.99.129) added. New cstate: current_term: 2 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:05:29.895622 16033 tablet_copy_client.cc:647] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:29.896782 16033 tablet_copy_client.cc:523] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:29.898355 16033 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:05:29.906809 16033 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:29.907074 16033 tablet_bootstrap.cc:492] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee: Bootstrap complete.
I20240503 14:05:29.907274 16033 ts_tablet_manager.cc:1367] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.001s
I20240503 14:05:29.907721 16033 raft_consensus.cc:348] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:05:29.907837 16033 raft_consensus.cc:729] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:05:29.907979 16033 consensus_queue.cc:260] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:05:29.908703 16033 ts_tablet_manager.cc:1398] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:05:29.909123 12057 tablet_copy_service.cc:342] P 623e8dcb58544e558207d8e652dfed5f: Request end of tablet copy session ca5386f84994496fa5376624391728ee-a4b4e16598b740cb9d690c2442a3e994 received from {username='slave'} at 127.0.99.131:56133
I20240503 14:05:29.909211 12057 tablet_copy_service.cc:433] P 623e8dcb58544e558207d8e652dfed5f: ending tablet copy session ca5386f84994496fa5376624391728ee-a4b4e16598b740cb9d690c2442a3e994 on tablet a4b4e16598b740cb9d690c2442a3e994 with peer ca5386f84994496fa5376624391728ee
I20240503 14:05:29.940297 15068 raft_consensus.cc:998] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee: Attempting to remove follower 1b4a61e8a11440e7a9f4b300d2442f75 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (300.164s)
I20240503 14:05:29.940719 15068 consensus_queue.cc:237] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:05:29.942452 12166 raft_consensus.cc:1261] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:29.943651 13775 raft_consensus.cc:1261] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:05:29.944087 14821 consensus_queue.cc:1035] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:05:29.944257 11882 catalog_manager.cc:4573] TS e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4224ca4d93ce4784b52a7550468ce1b1: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:29.946751 14821 consensus_queue.cc:1035] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:05:29.952384 14821 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.952540 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:29.953861 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.954274 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:29.954622 12166 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.954820 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:29.955291 11895 catalog_manager.cc:5484] T df13ec098dc548afac8c67195d25e7e8 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 6 to 7, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.959547 13775 raft_consensus.cc:2932] T df13ec098dc548afac8c67195d25e7e8 P 66190085d1c04e068d5c0dec9c4bf07d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } }
I20240503 14:05:29.962908 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:29.962917 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:30.062662 16087 ts_tablet_manager.cc:902] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d: Initiating tablet copy from peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:05:30.063163 16087 tablet_copy_client.cc:308] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36263
I20240503 14:05:30.066826 12188 tablet_copy_service.cc:140] P 8b17d706d2344dbaae13cda9b0733143: Received BeginTabletCopySession request for tablet 6c4f3bc64e7145e2b692beab730d1496 from peer 66190085d1c04e068d5c0dec9c4bf07d ({username='slave'} at 127.0.99.133:40923)
I20240503 14:05:30.066917 12188 tablet_copy_service.cc:161] P 8b17d706d2344dbaae13cda9b0733143: Beginning new tablet copy session on tablet 6c4f3bc64e7145e2b692beab730d1496 from peer 66190085d1c04e068d5c0dec9c4bf07d at {username='slave'} at 127.0.99.133:40923: session id = 66190085d1c04e068d5c0dec9c4bf07d-6c4f3bc64e7145e2b692beab730d1496
I20240503 14:05:30.067950 12188 tablet_copy_source_session.cc:205] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:05:30.068965 16087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4f3bc64e7145e2b692beab730d1496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:30.070856 16087 tablet_copy_client.cc:783] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 5 data blocks...
I20240503 14:05:30.075080 16087 tablet_copy_client.cc:647] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:30.076246 16087 tablet_copy_client.cc:523] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:30.077391 16087 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap starting.
I20240503 14:05:30.085027 16087 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:30.085224 16087 tablet_bootstrap.cc:492] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d: Bootstrap complete.
I20240503 14:05:30.085361 16087 ts_tablet_manager.cc:1367] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:05:30.085685 16087 raft_consensus.cc:348] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:05:30.085772 16087 raft_consensus.cc:729] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66190085d1c04e068d5c0dec9c4bf07d, State: Initialized, Role: FOLLOWER
I20240503 14:05:30.085865 16087 consensus_queue.cc:260] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:05:30.086118 16087 ts_tablet_manager.cc:1398] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:05:30.086546 12188 tablet_copy_service.cc:342] P 8b17d706d2344dbaae13cda9b0733143: Request end of tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-6c4f3bc64e7145e2b692beab730d1496 received from {username='slave'} at 127.0.99.133:40923
I20240503 14:05:30.086642 12188 tablet_copy_service.cc:433] P 8b17d706d2344dbaae13cda9b0733143: ending tablet copy session 66190085d1c04e068d5c0dec9c4bf07d-6c4f3bc64e7145e2b692beab730d1496 on tablet 6c4f3bc64e7145e2b692beab730d1496 with peer 66190085d1c04e068d5c0dec9c4bf07d
I20240503 14:05:30.109136 12166 raft_consensus.cc:1203] T 70ffc5888a5945d6a8e166420e6f1bd9 P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 14:05:30.113656 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(a4b4e16598b740cb9d690c2442a3e994): 4923 bytes on disk
I20240503 14:05:30.114073 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(a4b4e16598b740cb9d690c2442a3e994) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240503 14:05:30.156461 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:30.156770 15920 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (300.684s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:30.159008 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:05:30.160018 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.160240 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.161041 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:30.162268 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:30.171905 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:30.183766 15918 raft_consensus.cc:998] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143: Attempting to remove follower 1b4a61e8a11440e7a9f4b300d2442f75 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (300.417s)
I20240503 14:05:30.184146 15918 consensus_queue.cc:237] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:05:30.185303 12430 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:05:30.185910 13904 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:05:30.186062 16147 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:05:30.186496 16147 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:05:30.194296 16147 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:30.196923 11892 catalog_manager.cc:5484] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 8 to 9, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:30.197865 13904 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:30.198552 12430 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:30.243449 16190 ts_tablet_manager.cc:902] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:05:30.244302 16190 tablet_copy_client.cc:308] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:05:30.248521 12318 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet e92d8f55591346158b9583feb958a8ca from peer 8b17d706d2344dbaae13cda9b0733143 ({username='slave'} at 127.0.99.130:44445)
I20240503 14:05:30.248618 12318 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet e92d8f55591346158b9583feb958a8ca from peer 8b17d706d2344dbaae13cda9b0733143 at {username='slave'} at 127.0.99.130:44445: session id = 8b17d706d2344dbaae13cda9b0733143-e92d8f55591346158b9583feb958a8ca
I20240503 14:05:30.249900 12318 tablet_copy_source_session.cc:205] T e92d8f55591346158b9583feb958a8ca P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:05:30.251407 16190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92d8f55591346158b9583feb958a8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:30.253767 16190 tablet_copy_client.cc:783] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Starting download of 5 data blocks...
I20240503 14:05:30.258781 16190 tablet_copy_client.cc:647] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:30.260061 16190 tablet_copy_client.cc:523] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:30.262105 16190 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143: Bootstrap starting.
I20240503 14:05:30.272678 16190 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:30.272951 16190 tablet_bootstrap.cc:492] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143: Bootstrap complete.
I20240503 14:05:30.273113 16190 ts_tablet_manager.cc:1367] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240503 14:05:30.273481 16190 raft_consensus.cc:348] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:05:30.273583 16190 raft_consensus.cc:729] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b17d706d2344dbaae13cda9b0733143, State: Initialized, Role: FOLLOWER
I20240503 14:05:30.273721 16190 consensus_queue.cc:260] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } } peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } }
I20240503 14:05:30.273996 16190 ts_tablet_manager.cc:1398] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:05:30.274662 12318 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 8b17d706d2344dbaae13cda9b0733143-e92d8f55591346158b9583feb958a8ca received from {username='slave'} at 127.0.99.130:44445
I20240503 14:05:30.274736 12318 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 8b17d706d2344dbaae13cda9b0733143-e92d8f55591346158b9583feb958a8ca on tablet e92d8f55591346158b9583feb958a8ca with peer 8b17d706d2344dbaae13cda9b0733143
I20240503 14:05:30.288347 15920 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (300.819s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:30.297749 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.297907 13822 maintenance_manager.cc:419] P 66190085d1c04e068d5c0dec9c4bf07d: Scheduling UndoDeltaBlockGCOp(6c4f3bc64e7145e2b692beab730d1496): 4921 bytes on disk
I20240503 14:05:30.298329 13712 maintenance_manager.cc:643] P 66190085d1c04e068d5c0dec9c4bf07d: UndoDeltaBlockGCOp(6c4f3bc64e7145e2b692beab730d1496) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 14:05:30.305595 16237 ts_tablet_manager.cc:902] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
I20240503 14:05:30.306380 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:30.308326 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:30.309254 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:30.309748 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:30.315696 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:30.321763 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.323122 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:30.331250 16237 tablet_copy_client.cc:308] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:36399
I20240503 14:05:30.335386 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.338793 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.345021 12318 tablet_copy_service.cc:140] P ca5386f84994496fa5376624391728ee: Received BeginTabletCopySession request for tablet c8933689df2c45b59a9cd16024ddb7d9 from peer 623e8dcb58544e558207d8e652dfed5f ({username='slave'} at 127.0.99.129:34123)
I20240503 14:05:30.345135 12318 tablet_copy_service.cc:161] P ca5386f84994496fa5376624391728ee: Beginning new tablet copy session on tablet c8933689df2c45b59a9cd16024ddb7d9 from peer 623e8dcb58544e558207d8e652dfed5f at {username='slave'} at 127.0.99.129:34123: session id = 623e8dcb58544e558207d8e652dfed5f-c8933689df2c45b59a9cd16024ddb7d9
I20240503 14:05:30.346182 12318 tablet_copy_source_session.cc:205] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:05:30.353699 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:30.365543 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:30.366449 16237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8933689df2c45b59a9cd16024ddb7d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:30.377578 16237 tablet_copy_client.cc:783] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Starting download of 5 data blocks...
I20240503 14:05:30.388216 16237 tablet_copy_client.cc:647] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:30.390656 16237 tablet_copy_client.cc:523] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:30.392647 16237 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap starting.
I20240503 14:05:30.402510 16237 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:30.402778 16237 tablet_bootstrap.cc:492] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f: Bootstrap complete.
I20240503 14:05:30.402988 16237 ts_tablet_manager.cc:1367] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240503 14:05:30.403465 16237 raft_consensus.cc:348] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:05:30.403571 16237 raft_consensus.cc:729] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623e8dcb58544e558207d8e652dfed5f, State: Initialized, Role: FOLLOWER
I20240503 14:05:30.403721 16237 consensus_queue.cc:260] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:05:30.403990 16237 ts_tablet_manager.cc:1398] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:05:30.404390 12318 tablet_copy_service.cc:342] P ca5386f84994496fa5376624391728ee: Request end of tablet copy session 623e8dcb58544e558207d8e652dfed5f-c8933689df2c45b59a9cd16024ddb7d9 received from {username='slave'} at 127.0.99.129:34123
I20240503 14:05:30.404456 12318 tablet_copy_service.cc:433] P ca5386f84994496fa5376624391728ee: ending tablet copy session 623e8dcb58544e558207d8e652dfed5f-c8933689df2c45b59a9cd16024ddb7d9 on tablet c8933689df2c45b59a9cd16024ddb7d9 with peer 623e8dcb58544e558207d8e652dfed5f
I20240503 14:05:30.413342 12299 raft_consensus.cc:1203] T a4b4e16598b740cb9d690c2442a3e994 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 14:05:30.455368 13775 raft_consensus.cc:1203] T 6c4f3bc64e7145e2b692beab730d1496 P 66190085d1c04e068d5c0dec9c4bf07d [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.10->[3.11-3.11] Dedup: 3.11->[]
I20240503 14:05:30.466863 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:30.467819 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:30.470366 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.471318 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:30.480264 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.487399 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:30.492132 12084 maintenance_manager.cc:419] P 623e8dcb58544e558207d8e652dfed5f: Scheduling UndoDeltaBlockGCOp(c8933689df2c45b59a9cd16024ddb7d9): 4582 bytes on disk
I20240503 14:05:30.492624 11974 maintenance_manager.cc:643] P 623e8dcb58544e558207d8e652dfed5f: UndoDeltaBlockGCOp(c8933689df2c45b59a9cd16024ddb7d9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 14:05:30.501600 12215 maintenance_manager.cc:419] P 8b17d706d2344dbaae13cda9b0733143: Scheduling UndoDeltaBlockGCOp(e92d8f55591346158b9583feb958a8ca): 4922 bytes on disk
I20240503 14:05:30.502053 12105 maintenance_manager.cc:643] P 8b17d706d2344dbaae13cda9b0733143: UndoDeltaBlockGCOp(e92d8f55591346158b9583feb958a8ca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 14:05:30.555233 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (301.082s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:30.580531 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:30.581722 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:30.582738 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:30.588290 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.593693 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:30.608680 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.651917 12167 consensus_queue.cc:237] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } }
I20240503 14:05:30.653340 12430 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:05:30.653787 16149 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:05:30.655803 16147 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:30.656610 12430 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:30.659776 11891 catalog_manager.cc:5484] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 reported cstate change: config changed from index 9 to 10, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } }
I20240503 14:05:30.663914 12167 consensus_queue.cc:237] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:05:30.665439 12430 raft_consensus.cc:1261] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Refusing update from remote peer 8b17d706d2344dbaae13cda9b0733143: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 14:05:30.665879 12104 consensus_peers.cc:485] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 -> Peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399): Couldn't send request to peer ca5386f84994496fa5376624391728ee. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:30.666118 16149 consensus_queue.cc:1035] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:05:30.667126 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (301.198s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:30.669478 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 14:05:30.669642 13885 tablet_service.cc:1514] Processing DeleteTablet for tablet c42fee9fadbb4601a7fe32eb6a04e767 with delete_type TABLET_DATA_TOMBSTONED (TS b67f337e10994234baedbf0b19c13513 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35320
I20240503 14:05:30.669950 16147 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:05:30.671687 16395 tablet_replica.cc:330] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: stopping tablet replica
I20240503 14:05:30.672145 12430 raft_consensus.cc:2932] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:05:30.672823 16395 raft_consensus.cc:2229] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:30.673655 16395 raft_consensus.cc:2258] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:30.674369 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c42fee9fadbb4601a7fe32eb6a04e767 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:05:30.674871 11891 catalog_manager.cc:5484] T c42fee9fadbb4601a7fe32eb6a04e767 P 84c63c6f551949f6a1d7c07760824361 reported cstate change: config changed from index 10 to 11, VOTER ca5386f84994496fa5376624391728ee (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "8b17d706d2344dbaae13cda9b0733143" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } }
I20240503 14:05:30.677969 16395 ts_tablet_manager.cc:1875] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:30.681470 16395 ts_tablet_manager.cc:1888] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:05:30.681555 16395 log.cc:1196] T c42fee9fadbb4601a7fe32eb6a04e767 P b67f337e10994234baedbf0b19c13513: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/c42fee9fadbb4601a7fe32eb6a04e767
I20240503 14:05:30.682250 11881 catalog_manager.cc:4830] TS b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): tablet c42fee9fadbb4601a7fe32eb6a04e767 (table rebalance_test_table_4 [id=ba783d9b56f146b5b6b0e67c371ac98b]) successfully deleted
I20240503 14:05:30.705829 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:30.706101 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.709646 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:30.715816 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:30.720820 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:30.726796 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 14:05:30.741724 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.783118 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:30.787176 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:30.787292 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.788810 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:30.797158 12167 raft_consensus.cc:1203] T e92d8f55591346158b9583feb958a8ca P 8b17d706d2344dbaae13cda9b0733143 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 14:05:30.797459 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:30.812786 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.888146 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:30.895381 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:30.896291 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:30.896795 12037 raft_consensus.cc:1203] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:05:30.900269 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.903419 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:30.906812 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:31.030386 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:31.035387 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.035777 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.038622 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:31.050551 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:31.051741 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:31.069247 11881 catalog_manager.cc:4573] TS 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c8933689df2c45b59a9cd16024ddb7d9: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:31.090936 15920 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (301.622s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:31.095929 12299 consensus_queue.cc:237] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:05:31.096911 12037 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:05:31.097270 15920 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:05:31.098788 14821 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:05:31.099193 12037 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:05:31.102224 11888 catalog_manager.cc:5484] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 8 to 9, VOTER 84c63c6f551949f6a1d7c07760824361 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } }
I20240503 14:05:31.105370 12408 tablet_service.cc:1514] Processing DeleteTablet for tablet c8933689df2c45b59a9cd16024ddb7d9 with delete_type TABLET_DATA_TOMBSTONED (TS 84c63c6f551949f6a1d7c07760824361 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42886
I20240503 14:05:31.105711 16552 tablet_replica.cc:330] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361: stopping tablet replica
I20240503 14:05:31.105880 16552 raft_consensus.cc:2229] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:31.106056 16552 raft_consensus.cc:2258] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:31.106097 16553 ts_tablet_manager.cc:902] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Initiating tablet copy from peer 8b17d706d2344dbaae13cda9b0733143 (127.0.99.130:36263)
I20240503 14:05:31.106220 12298 consensus_queue.cc:237] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } }
I20240503 14:05:31.106639 16552 ts_tablet_manager.cc:1875] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:31.106961 16553 tablet_copy_client.cc:308] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:36263
W20240503 14:05:31.107511 12234 consensus_peers.cc:485] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8933689df2c45b59a9cd16024ddb7d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:31.107664 12188 tablet_copy_service.cc:140] P 8b17d706d2344dbaae13cda9b0733143: Received BeginTabletCopySession request for tablet c42fee9fadbb4601a7fe32eb6a04e767 from peer ca5386f84994496fa5376624391728ee ({username='slave'} at 127.0.99.131:38211)
I20240503 14:05:31.107769 12188 tablet_copy_service.cc:161] P 8b17d706d2344dbaae13cda9b0733143: Beginning new tablet copy session on tablet c42fee9fadbb4601a7fe32eb6a04e767 from peer ca5386f84994496fa5376624391728ee at {username='slave'} at 127.0.99.131:38211: session id = ca5386f84994496fa5376624391728ee-c42fee9fadbb4601a7fe32eb6a04e767
I20240503 14:05:31.108620 12188 tablet_copy_source_session.cc:205] T c42fee9fadbb4601a7fe32eb6a04e767 P 8b17d706d2344dbaae13cda9b0733143: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:05:31.109350 16553 ts_tablet_manager.cc:1875] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:05:31.109527 12037 raft_consensus.cc:1261] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Refusing update from remote peer ca5386f84994496fa5376624391728ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:05:31.109895 15920 consensus_queue.cc:1035] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:05:31.109977 16552 ts_tablet_manager.cc:1888] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240503 14:05:31.110078 16552 log.cc:1196] T c8933689df2c45b59a9cd16024ddb7d9 P 84c63c6f551949f6a1d7c07760824361: Deleting WAL directory at /tmp/dist-test-taskGjG1E4/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714744717693984-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c8933689df2c45b59a9cd16024ddb7d9
I20240503 14:05:31.110616 11879 catalog_manager.cc:4830] TS 84c63c6f551949f6a1d7c07760824361 (127.0.99.132:33927): tablet c8933689df2c45b59a9cd16024ddb7d9 (table rebalance_test_table_6 [id=fe2fab20a73a4d2696ed4dbcc8243c72]) successfully deleted
I20240503 14:05:31.111315 14821 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:31.111585 16553 ts_tablet_manager.cc:1888] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 14:05:31.111743 16553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42fee9fadbb4601a7fe32eb6a04e767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:31.111733 12037 raft_consensus.cc:2932] T c8933689df2c45b59a9cd16024ddb7d9 P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:31.112705 16553 tablet_copy_client.cc:783] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: tablet copy: Starting download of 5 data blocks...
I20240503 14:05:31.112690 11882 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c8933689df2c45b59a9cd16024ddb7d9 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:05:31.113759 11888 catalog_manager.cc:5484] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee reported cstate change: config changed from index 9 to 10, VOTER b67f337e10994234baedbf0b19c13513 (127.0.99.134) added. New cstate: current_term: 2 leader_uuid: "ca5386f84994496fa5376624391728ee" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } } peers { permanent_uuid: "b67f337e10994234baedbf0b19c13513" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 37075 } } }
I20240503 14:05:31.117424 16553 tablet_copy_client.cc:647] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:31.118552 16553 tablet_copy_client.cc:523] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:31.119526 16553 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Bootstrap starting.
I20240503 14:05:31.126133 16553 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:31.126363 16553 tablet_bootstrap.cc:492] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Bootstrap complete.
I20240503 14:05:31.126534 16553 ts_tablet_manager.cc:1367] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:05:31.126856 16553 raft_consensus.cc:348] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:05:31.127003 16553 raft_consensus.cc:729] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5386f84994496fa5376624391728ee, State: Initialized, Role: FOLLOWER
I20240503 14:05:31.127125 16553 consensus_queue.cc:260] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } }
I20240503 14:05:31.127411 16553 ts_tablet_manager.cc:1398] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:05:31.127894 12188 tablet_copy_service.cc:342] P 8b17d706d2344dbaae13cda9b0733143: Request end of tablet copy session ca5386f84994496fa5376624391728ee-c42fee9fadbb4601a7fe32eb6a04e767 received from {username='slave'} at 127.0.99.131:38211
I20240503 14:05:31.127969 12188 tablet_copy_service.cc:433] P 8b17d706d2344dbaae13cda9b0733143: ending tablet copy session ca5386f84994496fa5376624391728ee-c42fee9fadbb4601a7fe32eb6a04e767 on tablet c42fee9fadbb4601a7fe32eb6a04e767 with peer ca5386f84994496fa5376624391728ee
I20240503 14:05:31.168757 15920 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (301.696s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:31.209096 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:31.209389 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.210115 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:31.213315 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.222700 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:31.234165 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:31.319018 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.323834 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.327425 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:31.328270 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:31.330554 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:31.331158 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:31.368412 12346 maintenance_manager.cc:419] P ca5386f84994496fa5376624391728ee: Scheduling UndoDeltaBlockGCOp(c42fee9fadbb4601a7fe32eb6a04e767): 4923 bytes on disk
I20240503 14:05:31.368906 12236 maintenance_manager.cc:643] P ca5386f84994496fa5376624391728ee: UndoDeltaBlockGCOp(c42fee9fadbb4601a7fe32eb6a04e767) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240503 14:05:31.403122 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:31.409713 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:31.414134 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:31.415524 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.415562 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.418023 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:31.479156 14220 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskGjG1E4/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:00:14.124635 14226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:00:14.125145 14226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:00:14.129544 14226 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:00:15.215900 14226 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:00:15.833076 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.833276 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.833388 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.833527 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.833598 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.833638 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.833671 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.833736 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.833787 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.833858 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:15.833915 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.833976 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:15.834017 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:15.834049 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:15.834082 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:15.890172 14226 rebalancer_tool.cc:1074] tablet 5e6ca408c4254678b67a74f49a34881b: '623e8dcb58544e558207d8e652dfed5f' -> '66190085d1c04e068d5c0dec9c4bf07d' move scheduled
I20240503 14:00:16.007418 14226 rebalancer_tool.cc:1074] tablet a0106f7a40cb467eb4738890ac4e22d4: '8b17d706d2344dbaae13cda9b0733143' -> '1b4a61e8a11440e7a9f4b300d2442f75' move scheduled
I20240503 14:00:16.207420 14226 rebalancer_tool.cc:1074] tablet 7d2e1517a8ac4041bfb4b90634cb0fa6: '84c63c6f551949f6a1d7c07760824361' -> 'b67f337e10994234baedbf0b19c13513' move scheduled
I20240503 14:00:16.261238 14226 rebalancer_tool.cc:1074] tablet 11a8c270532947ba9b505aace44b56f6: '623e8dcb58544e558207d8e652dfed5f' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move scheduled
I20240503 14:00:16.327751 14226 rebalancer_tool.cc:1074] tablet 4a608f683c4f41f3bbdfc9b4c219ce49: 'ca5386f84994496fa5376624391728ee' -> '66190085d1c04e068d5c0dec9c4bf07d' move scheduled
I20240503 14:00:16.378969 14226 rebalancer_tool.cc:1084] tablet a6a89230883347d4a64d96d321247012: '8b17d706d2344dbaae13cda9b0733143' -> '1b4a61e8a11440e7a9f4b300d2442f75' move ignored: Illegal state: tablet a6a89230883347d4a64d96d321247012: replica 1b4a61e8a11440e7a9f4b300d2442f75 is already present
I20240503 14:00:16.379072 14226 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:16.790520 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.791085 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.791144 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.791174 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.791215 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.791285 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.791330 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.791357 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.791401 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.791455 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.791489 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ef1de2e06d4947918f6cc1607fffc4ee: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.791523 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.791563 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.791599 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 40371d9909ff4960ac96d09aac7936cc: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.791628 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cf9cc41812f545f6bd3749ad1d6873a8: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.791664 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 22cca43e02d54fad8bd2569ae6b8bd0f: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:16.791702 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.791777 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:16.791826 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:16.791872 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:16.791909 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ba783d9b56f146b5b6b0e67c371ac98b: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
I20240503 14:00:17.062903 14226 rebalancer_tool.cc:1074] tablet 211de6f35a844e278e8c1eb1621e7c5f: '84c63c6f551949f6a1d7c07760824361' -> '1b4a61e8a11440e7a9f4b300d2442f75' move scheduled
I20240503 14:00:17.296854 14226 rebalancer_tool.cc:1074] tablet 70ffc5888a5945d6a8e166420e6f1bd9: 'ca5386f84994496fa5376624391728ee' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move scheduled
I20240503 14:00:17.364557 14226 rebalancer_tool.cc:1074] tablet 0e442bb664f44887aa8afdb62e1d97fc: '84c63c6f551949f6a1d7c07760824361' -> 'b67f337e10994234baedbf0b19c13513' move scheduled
I20240503 14:00:17.378209 14226 rebalancer_tool.cc:1084] tablet b031f09b97664d1a94c14f0661181899: '8b17d706d2344dbaae13cda9b0733143' -> '1b4a61e8a11440e7a9f4b300d2442f75' move ignored: Illegal state: tablet b031f09b97664d1a94c14f0661181899: replica 1b4a61e8a11440e7a9f4b300d2442f75 is already present
I20240503 14:00:17.378298 14226 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:00:17.856588 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server e85b0283a0fd4e3ba81b4170f3cdacb5
W20240503 14:00:17.856781 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 14df5f9889f146058ff6837fc3ab4239: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server 66190085d1c04e068d5c0dec9c4bf07d
W20240503 14:00:17.856819 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f77268cca2684060a44e936632d05050: could not find any suitable replica to move from server 84c63c6f551949f6a1d7c07760824361 to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.856861 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e37918da7d2407cbb2fc828ae4e301b: could not find any suitable replica to move from server 8b17d706d2344dbaae13cda9b0733143 to server 1b4a61e8a11440e7a9f4b300d2442f75
W20240503 14:00:17.856894 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fbf10f839f48498b898afc6af4a813a6: could not find any suitable replica to move from server ca5386f84994496fa5376624391728ee to server b67f337e10994234baedbf0b19c13513
W20240503 14:00:17.858093 14226 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fe2fab20a73a4d2696ed4dbcc8243c72: could not find any suitable replica to move from server 623e8dcb58544e558207d8e652dfed5f to server 1b4a61e8a11440e7a9f4b300d2442f75
I20240503 14:00:17.994588 14226 rebalancer_tool.cc:1074] tablet a6a89230883347d4a64d96d321247012: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:18.043586 14226 rebalancer_tool.cc:1074] tablet f4f9cb87c70e43dbb0d850c275ebb8a5: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:18.115676 14226 rebalancer_tool.cc:1074] tablet adf2c44328e244779955e712b99f8331: '623e8dcb58544e558207d8e652dfed5f' -> 'b67f337e10994234baedbf0b19c13513' move scheduled
I20240503 14:00:18.258468 14226 rebalancer_tool.cc:1074] tablet df13ec098dc548afac8c67195d25e7e8: '84c63c6f551949f6a1d7c07760824361' -> '66190085d1c04e068d5c0dec9c4bf07d' move scheduled
I20240503 14:00:18.387403 14226 rebalancer_tool.cc:1074] tablet 3481715d755a4277b0905b0832fad4af: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:18.420533 14226 rebalancer_tool.cc:1074] tablet d837845c6f644bca83315c9f7c45246d: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:18.450541 14226 rebalancer_tool.cc:1074] tablet 79b21b1e5292488aad442060b417f3c1: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:22.937983 14226 rebalancer_tool.cc:1136] tablet 7d2e1517a8ac4041bfb4b90634cb0fa6: '84c63c6f551949f6a1d7c07760824361' -> 'b67f337e10994234baedbf0b19c13513' move completed: OK
I20240503 14:00:22.957621 14226 rebalancer_tool.cc:1136] tablet d837845c6f644bca83315c9f7c45246d: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:23.743731 14226 rebalancer_tool.cc:1136] tablet 11a8c270532947ba9b505aace44b56f6: '623e8dcb58544e558207d8e652dfed5f' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move completed: OK
I20240503 14:00:23.750684 14226 rebalancer_tool.cc:1136] tablet 79b21b1e5292488aad442060b417f3c1: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:23.759348 14226 rebalancer_tool.cc:1136] tablet a6a89230883347d4a64d96d321247012: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
I20240503 14:00:23.778246 14226 rebalancer_tool.cc:1136] tablet 4a608f683c4f41f3bbdfc9b4c219ce49: 'ca5386f84994496fa5376624391728ee' -> '66190085d1c04e068d5c0dec9c4bf07d' move completed: OK
I20240503 14:00:24.723459 14226 rebalancer_tool.cc:1074] tablet e92d8f55591346158b9583feb958a8ca: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:24.753484 14226 rebalancer_tool.cc:1074] tablet c2e9f6eecbec45988b290775277dbddc: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:24.786142 14226 rebalancer_tool.cc:1074] tablet f5c06806a19e4f4592d26ed4acc651cd: 'ca5386f84994496fa5376624391728ee' -> '' move scheduled
I20240503 14:00:24.798048 14226 rebalancer_tool.cc:1136] tablet e92d8f55591346158b9583feb958a8ca: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
I20240503 14:00:25.244700 14226 rebalancer_tool.cc:1123] tablet df13ec098dc548afac8c67195d25e7e8: '84c63c6f551949f6a1d7c07760824361' -> '66190085d1c04e068d5c0dec9c4bf07d' move is abandoned: Not found: Server with UUID 84c63c6f551949f6a1d7c07760824361 not a member of the config. RaftConfig: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "ca5386f84994496fa5376624391728ee" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 36399 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "66190085d1c04e068d5c0dec9c4bf07d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37005 } }
I20240503 14:00:25.254281 14226 rebalancer_tool.cc:1136] tablet f4f9cb87c70e43dbb0d850c275ebb8a5: 'ca5386f84994496fa5376624391728ee' -> '' move completed: OK
I20240503 14:00:25.309689 14226 rebalancer_tool.cc:1136] tablet 0e442bb664f44887aa8afdb62e1d97fc: '84c63c6f551949f6a1d7c07760824361' -> 'b67f337e10994234baedbf0b19c13513' move completed: OK
I20240503 14:00:25.321574 14226 rebalancer_tool.cc:1136] tablet 3481715d755a4277b0905b0832fad4af: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
I20240503 14:00:26.181447 14226 rebalancer_tool.cc:1136] tablet 5e6ca408c4254678b67a74f49a34881b: '623e8dcb58544e558207d8e652dfed5f' -> '66190085d1c04e068d5c0dec9c4bf07d' move completed: OK
I20240503 14:00:26.273984 14226 rebalancer_tool.cc:1136] tablet c2e9f6eecbec45988b290775277dbddc: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
I20240503 14:00:26.296634 14226 rebalancer_tool.cc:1136] tablet adf2c44328e244779955e712b99f8331: '623e8dcb58544e558207d8e652dfed5f' -> 'b67f337e10994234baedbf0b19c13513' move completed: OK
I20240503 14:00:26.813946 14226 rebalancer_tool.cc:1136] tablet 70ffc5888a5945d6a8e166420e6f1bd9: 'ca5386f84994496fa5376624391728ee' -> 'e85b0283a0fd4e3ba81b4170f3cdacb5' move completed: OK
I20240503 14:00:27.639866 14226 rebalancer_tool.cc:1074] tablet 2089eeda1e0f4ff5926584d472e6631d: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:27.651564 14226 rebalancer_tool.cc:1074] tablet a4b4e16598b740cb9d690c2442a3e994: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:27.737752 14226 rebalancer_tool.cc:1074] tablet 92506cde7b084b14b88d4a915731eeb3: '623e8dcb58544e558207d8e652dfed5f' -> '' move scheduled
I20240503 14:00:27.756191 14226 rebalancer_tool.cc:1074] tablet c8933689df2c45b59a9cd16024ddb7d9: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:28.970985 14226 rebalancer_tool.cc:1136] tablet 92506cde7b084b14b88d4a915731eeb3: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
W20240503 14:00:29.726732 22995 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:29.504636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:46845 (local address 127.0.0.1:40546)
0503 14:00:29.555670 (+ 51034us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:29.555684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:29.555759 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:29.726018 (+170259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":50963}
I20240503 14:00:29.816546 14226 rebalancer_tool.cc:1136] tablet a0106f7a40cb467eb4738890ac4e22d4: '8b17d706d2344dbaae13cda9b0733143' -> '1b4a61e8a11440e7a9f4b300d2442f75' move completed: OK
I20240503 14:00:29.831859 14226 rebalancer_tool.cc:1123] tablet f5c06806a19e4f4592d26ed4acc651cd: 'ca5386f84994496fa5376624391728ee' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:00:29.838021 14226 rebalancer_tool.cc:1136] tablet 211de6f35a844e278e8c1eb1621e7c5f: '84c63c6f551949f6a1d7c07760824361' -> '1b4a61e8a11440e7a9f4b300d2442f75' move completed: OK
I20240503 14:00:29.843436 14226 rebalancer_tool.cc:1074] tablet dcdc3ae5232a49cdb003d9a2e3cf6d92: '8b17d706d2344dbaae13cda9b0733143' -> '' move scheduled
I20240503 14:00:29.849524 14226 rebalancer_tool.cc:1136] tablet dcdc3ae5232a49cdb003d9a2e3cf6d92: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
W20240503 14:00:30.165967 23231 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:00:29.946951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:34763 (local address 127.0.0.1:53226)
0503 14:00:29.947142 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0503 14:00:29.947157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 14:00:29.947238 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:00:30.165847 (+218609us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":106}
I20240503 14:00:30.223497 14226 rebalancer_tool.cc:1136] tablet a4b4e16598b740cb9d690c2442a3e994: '8b17d706d2344dbaae13cda9b0733143' -> '' move completed: OK
I20240503 14:00:30.766419 14226 rebalancer_tool.cc:1074] tablet d1e9155230c74e92b6ee441de713a419: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:30.786902 14226 rebalancer_tool.cc:1074] tablet faa0a6ca00d442eaaf9fca7fc1259ec7: '84c63c6f551949f6a1d7c07760824361' -> '' move scheduled
I20240503 14:00:31.051091 14226 rebalancer_tool.cc:1136] tablet d1e9155230c74e92b6ee441de713a419: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:00:31.885330 14226 rebalancer_tool.cc:1136] tablet 2089eeda1e0f4ff5926584d472e6631d: '623e8dcb58544e558207d8e652dfed5f' -> '' move completed: OK
W20240503 14:03:49.208546 6566 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.000s sys 0.000s
I20240503 14:05:29.660778 14226 rebalancer_tool.cc:1136] tablet faa0a6ca00d442eaaf9fca7fc1259ec7: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
I20240503 14:05:31.311484 14226 rebalancer_tool.cc:1136] tablet c8933689df2c45b59a9cd16024ddb7d9: '84c63c6f551949f6a1d7c07760824361' -> '' move completed: OK
Illegal state: tablet server 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): unacceptable health status UNAVAILABLE
I20240503 14:05:31.506455 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.506752 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:31.507771 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:31.517572 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:05:31.519412 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.533578 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:31.573671 16754 ts_tablet_manager.cc:902] T c8933689df2c45b59a9cd16024ddb7d9 P b67f337e10994234baedbf0b19c13513: Initiating tablet copy from peer ca5386f84994496fa5376624391728ee (127.0.99.131:36399)
F20240503 14:05:31.573872 16754 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.queue_time_us, tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.queue_time_us) (tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.run_cpu_time_us, tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.run_cpu_time_us) (tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.run_wall_time_us, tablet-download-pool-0e442bb664f44887aa8afdb62e1d97fc.run_wall_time_us) (tablet-download-pool-2089eeda1e0f4ff5926584d472e6631d.queue_time_us, tablet-download-pool-2089eeda1e0f4ff5926584d472e6631d.queue_time_us) (tablet-download-pool-2089eeda1e0f4ff5926584d472e6631d.run_cpu_time_us, tablet-download-pool-2089eeda1e0f4ff5926584d472e6631d.run_cpu_time_us) (tablet-download-pool-2089eeda1e0f4ff5926584d472e6631d.run_wall_time_us, tablet-download-pool-2089eeda1e0f4ff5926584d472e6631d.run_wall_time_us) (tablet-download-pool-355b5456887b4c5694badc9d53b1809a.queue_time_us, tablet-download-pool-355b5456887b4c5694badc9d53b1809a.queue_time_us) (tablet-download-pool-355b5456887b4c5694badc9d53b1809a.run_cpu_time_us, tablet-download-pool-355b5456887b4c5694badc9d53b1809a.run_cpu_time_us) (tablet-download-pool-355b5456887b4c5694badc9d53b1809a.run_wall_time_us, tablet-download-pool-355b5456887b4c5694badc9d53b1809a.run_wall_time_us) (tablet-download-pool-5e6ca408c4254678b67a74f49a34881b.queue_time_us, tablet-download-pool-5e6ca408c4254678b67a74f49a34881b.queue_time_us) (tablet-download-pool-5e6ca408c4254678b67a74f49a34881b.run_cpu_time_us, tablet-download-pool-5e6ca408c4254678b67a74f49a34881b.run_cpu_time_us) (tablet-download-pool-5e6ca408c4254678b67a74f49a34881b.run_wall_time_us, tablet-download-pool-5e6ca408c4254678b67a74f49a34881b.run_wall_time_us) (tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.queue_time_us, tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.queue_time_us) (tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.run_cpu_time_us, tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.run_cpu_time_us) (tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.run_wall_time_us, tablet-download-pool-6c4f3bc64e7145e2b692beab730d1496.run_wall_time_us) (tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.queue_time_us, tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.queue_time_us) (tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.run_cpu_time_us, tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.run_cpu_time_us) (tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.run_wall_time_us, tablet-download-pool-70ffc5888a5945d6a8e166420e6f1bd9.run_wall_time_us) (tablet-download-pool-7d2e1517a8ac4041bfb4b90634cb0fa6.queue_time_us, tablet-download-pool-7d2e1517a8ac4041bfb4b90634cb0fa6.queue_time_us) (tablet-download-pool-7d2e1517a8ac4041bfb4b90634cb0fa6.run_cpu_time_us, tablet-download-pool-7d2e1517a8ac4041bfb4b90634cb0fa6.run_cpu_time_us) (tablet-download-pool-7d2e1517a8ac4041bfb4b90634cb0fa6.run_wall_time_us, tablet-download-pool-7d2e1517a8ac4041bfb4b90634cb0fa6.run_wall_time_us) (tablet-download-pool-92506cde7b084b14b88d4a915731eeb3.queue_time_us, tablet-download-pool-92506cde7b084b14b88d4a915731eeb3.queue_time_us) (tablet-download-pool-92506cde7b084b14b88d4a915731eeb3.run_cpu_time_us, tablet-download-pool-92506cde7b084b14b88d4a915731eeb3.run_cpu_time_us) (tablet-download-pool-92506cde7b084b14b88d4a915731eeb3.run_wall_time_us, tablet-download-pool-92506cde7b084b14b88d4a915731eeb3.run_wall_time_us) (tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.queue_time_us, tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.queue_time_us) (tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_cpu_time_us, tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_cpu_time_us) (tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_wall_time_us, tablet-download-pool-a0106f7a40cb467eb4738890ac4e22d4.run_wall_time_us) (tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.queue_time_us, tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.queue_time_us) (tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.run_cpu_time_us, tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.run_cpu_time_us) (tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.run_wall_time_us, tablet-download-pool-a4b4e16598b740cb9d690c2442a3e994.run_wall_time_us) (tablet-download-pool-adf2c44328e244779955e712b99f8331.queue_time_us, tablet-download-pool-adf2c44328e244779955e712b99f8331.queue_time_us) (tablet-download-pool-adf2c44328e244779955e712b99f8331.run_cpu_time_us, tablet-download-pool-adf2c44328e244779955e712b99f8331.run_cpu_time_us) (tablet-download-pool-adf2c44328e244779955e712b99f8331.run_wall_time_us, tablet-download-pool-adf2c44328e244779955e712b99f8331.run_wall_time_us) (tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.queue_time_us, tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.queue_time_us) (tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.run_cpu_time_us, tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.run_cpu_time_us) (tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.run_wall_time_us, tablet-download-pool-c42fee9fadbb4601a7fe32eb6a04e767.run_wall_time_us) (tablet-download-pool-c8933689df2c45b59a9cd16024ddb7d9.queue_time_us, tablet-download-pool-c8933689df2c45b59a9cd16024ddb7d9.queue_time_us) (tablet-download-pool-dcdc3ae5232a49cdb003d9a2e3cf6d92.queue_time_us, tablet-download-pool-dcdc3ae5232a49cdb003d9a2e3cf6d92.queue_time_us) (tablet-download-pool-dcdc3ae5232a49cdb003d9a2e3cf6d92.run_cpu_time_us, tablet-download-pool-dcdc3ae5232a49cdb003d9a2e3cf6d92.run_cpu_time_us) (tablet-download-pool-dcdc3ae5232a49cdb003d9a2e3cf6d92.run_wall_time_us, tablet-download-pool-dcdc3ae5232a49cdb003d9a2e3cf6d92.run_wall_time_us) (tablet-download-pool-faa0a6ca00d442eaaf9fca7fc1259ec7.queue_time_us, tablet-download-pool-faa0a6ca00d442eaaf9fca7fc1259ec7.queue_time_us) (tablet-download-pool-faa0a6ca00d442eaaf9fca7fc1259ec7.run_cpu_time_us, tablet-download-pool-faa0a6ca00d442eaaf9fca7fc1259ec7.run_cpu_time_us) (tablet-download-pool-faa0a6ca00d442eaaf9fca7fc1259ec7.run_wall_time_us, tablet-download-pool-faa0a6ca00d442eaaf9fca7fc1259ec7.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f229160cdcd google::LogMessage::Fail() at ??:0
@ 0x7f2291610b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f229160c7cc google::LogMessage::Flush() at ??:0
@ 0x7f229160df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2292a76044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f2292a5af98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f2292a58efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f229560b77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f2295614a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f229569576f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f22956a94f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f2295694226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 14:05:31.622380 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:31.625015 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.626667 15920 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (302.157s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:31.632951 13885 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7f22956a3c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561c2cb4575c std::function<>::operator()() at ??:0
I20240503 14:05:31.634690 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:31.636200 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
@ 0x7f2292a5de06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2292a5e6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f2292a6003b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561c2cb4575c std::function<>::operator()() at ??:0
@ 0x7f2292a4d807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f229328b6db start_thread at ??:0
@ 0x7f22907c371f clone at ??:0
W20240503 14:05:31.654651 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:05:31.658602 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:31.698868 12298 raft_consensus.cc:1203] T c42fee9fadbb4601a7fe32eb6a04e767 P ca5386f84994496fa5376624391728ee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240503 14:05:31.720546 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 14:05:31.721269 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:31.743186 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:05:31.751264 15920 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (302.279s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:31.758327 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:31.816318 12365 connection.cc:535] client connection to 127.0.99.134:37075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:05:31.816344 12234 connection.cc:535] client connection to 127.0.99.134:37075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:05:31.816308 13710 connection.cc:535] client connection to 127.0.99.134:37075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:05:31.816478 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:05:31.817240 12103 connection.cc:535] client connection to 127.0.99.134:37075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:05:31.817323 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 21 similar messages]
W20240503 14:05:31.817466 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:31.817530 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:31.817633 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:31.817669 11972 connection.cc:535] client connection to 127.0.99.134:37075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:05:31.818238 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:31.818396 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:31.818480 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:31.818806 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:31.897102 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:31.922726 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:31.957511 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:31.963618 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:32.015282 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:32.025458 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:32.032616 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:32.036016 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:32.047123 15920 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (302.578s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:32.079255 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:32.083786 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:32.085711 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:32.087777 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:32.093617 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:32.100966 11881 catalog_manager.cc:4573] TS 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c8933689df2c45b59a9cd16024ddb7d9: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:32.106477 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:32.160594 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:32.172432 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:32.175908 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:32.177291 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:32.184741 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:32.196894 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:32.198408 15920 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (302.726s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:32.405627 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:32.474159 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:05:32.475046 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (303.006s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:32.476428 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:32.477578 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:32.477627 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:32.479496 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:32.484421 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:32.564607 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:32.565004 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:32.572409 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:32.575196 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:32.576256 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:32.602691 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (303.130s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:32.603463 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 14:05:32.867192 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:32.869614 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:32.871752 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:32.872184 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:32.879132 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:32.890218 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:32.961371 11884 maintenance_manager.cc:419] P e301624eacf4431890d4c35d6eed9916: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7533 bytes on disk
I20240503 14:05:32.966635 11883 maintenance_manager.cc:643] P e301624eacf4431890d4c35d6eed9916: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.001s sys 0.001s Metrics: {"dirs.queue_time_us":144,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":2826}
I20240503 14:05:32.977758 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:32.978209 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:32.984186 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:32.985483 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:33.002199 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:33.014413 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (303.542s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:33.059957 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (303.591s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:33.060752 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 14:05:33.253607 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:33.257570 11882 catalog_manager.cc:4573] TS e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet faa0a6ca00d442eaaf9fca7fc1259ec7: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:33.283308 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:33.286538 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:33.287686 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:33.290871 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:33.300365 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:33.381843 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:33.385799 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:33.386096 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:33.386631 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:33.388635 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:33.457415 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (303.988s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:33.537551 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (304.065s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:33.682778 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:33.683694 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:33.684628 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:33.687927 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:33.692055 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:33.710233 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:33.776829 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:33.777339 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:33.778777 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:33.780798 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:33.793526 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:33.969514 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (304.500s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:33.985616 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (304.513s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:34.006613 11881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 166 similar messages]
W20240503 14:05:34.038002 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:05:34.071342 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:34.073063 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:34.073822 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:34.075935 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:34.083088 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:34.147027 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:34.164693 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:05:34.167169 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:34.171756 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:34.173233 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:34.176376 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:34.178431 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:34.205089 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:34.211539 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:34.212076 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:34.223706 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:34.225312 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:34.229043 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:34.260432 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 14:05:34.260982 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:34.270748 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:34.284811 11881 catalog_manager.cc:4573] TS 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet df13ec098dc548afac8c67195d25e7e8: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:34.293179 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:34.332247 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:34.357973 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:34.373487 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:05:34.394645 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (304.925s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:34.409828 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:05:34.467832 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:34.470091 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:34.472182 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:34.474121 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:34.479009 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:34.480836 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:34.519929 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:05:34.527403 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (305.055s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:34.563637 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:34.565654 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:34.567162 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:34.571550 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:34.583488 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:34.640563 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:34.716775 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:05:34.848904 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:34.859154 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:34.865254 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:34.866324 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:34.866920 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:34.876288 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:34.899143 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:05:34.912441 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (305.440s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:34.913326 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 14:05:34.963584 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:34.965396 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:34.965641 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:34.966531 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:34.976056 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:35.012444 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (305.543s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:35.266788 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:35.267259 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:35.268867 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:35.270661 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:35.275974 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:35.331573 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:35.363257 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:35.364925 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:35.365043 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:35.367393 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:35.370482 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:35.443405 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (305.971s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:35.467430 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (305.998s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:35.468266 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240503 14:05:35.656562 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:35.658545 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:35.659336 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:35.673036 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:35.680069 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:35.762090 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:35.764559 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:35.764654 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:35.765722 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:35.772068 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:35.868620 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (306.396s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:35.910269 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:35.912968 16396 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:05:35.913095 16396 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:05:35.913549 16396 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:05:35.914335 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:35.914418 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:35.914492 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:35.914630 16396 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:05:35.973850 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 17 similar messages]
I20240503 14:05:36.065086 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:36.069595 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:36.069646 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (306.600s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:36.069870 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.069906 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:36.069931 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.176729 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:36.177402 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:36.179898 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.190717 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.195044 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:36.391513 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:05:36.466583 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (306.994s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:36.477550 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:36.495905 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:36.496676 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:36.500716 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:36.507745 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.510752 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.587895 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:36.591878 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.592836 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.595357 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:36.597411 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:36.606458 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:36.608003 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:36.611805 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:05:36.629031 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (307.160s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:36.632082 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:36.644433 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:36.740662 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 14:05:36.741286 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:36.754320 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:36.766338 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:36.804093 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:36.832026 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:36.862841 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:36.863329 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:05:36.888382 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:36.891045 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.892969 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:36.897822 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.908185 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:36.935616 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:36.975553 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:05:36.981655 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (307.509s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:36.991961 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.995420 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:36.995440 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:36.996047 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:36.997041 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:37.001785 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 14:05:37.038774 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:37.054091 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:37.059464 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:05:37.059935 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:37.125083 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:05:37.223811 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (307.755s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:37.292415 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:37.292635 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:37.297562 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:37.303575 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:37.306107 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:37.384006 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:37.385432 11882 catalog_manager.cc:4573] TS e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet faa0a6ca00d442eaaf9fca7fc1259ec7: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:37.386895 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:37.387645 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:37.388025 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:37.388163 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:37.441028 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (307.968s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:37.442080 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 14:05:37.461251 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:37.474784 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:05:37.570923 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:37.685945 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:37.689170 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:37.691322 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:37.695365 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:37.702958 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:37.777549 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (308.308s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:37.786567 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:37.790519 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:37.797705 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:37.800793 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:37.806434 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:37.873798 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (308.401s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:38.000310 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:38.091746 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:38.092940 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:38.099887 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:38.099979 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:38.100346 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:38.177472 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (308.708s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:38.178507 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240503 14:05:38.189204 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:38.191395 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:38.192420 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:38.194171 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:38.194207 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:38.394668 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (308.922s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:38.406090 11881 catalog_manager.cc:4573] TS 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet df13ec098dc548afac8c67195d25e7e8: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:38.489589 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:38.490522 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:38.496085 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:38.503677 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:38.507328 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:38.508577 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:38.586027 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:38.592485 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:38.592543 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:38.603224 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:38.608116 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:38.618058 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (309.149s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:38.795998 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:38.809302 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:05:38.887143 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:38.889451 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:38.891607 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:38.894846 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (309.422s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:38.897172 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:38.898209 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:38.922283 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:38.969605 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:05:38.985060 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:38.986732 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:38.988062 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:38.991312 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:38.997820 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (309.529s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:39.003497 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:39.078511 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:39.081914 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:39.098451 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:39.152190 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:39.165670 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:05:39.280478 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:05:39.285704 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:05:39.287508 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:39.288153 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:39.290057 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:39.297433 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:39.299705 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:39.301759 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:39.311615 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:05:39.313650 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (309.841s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:39.376048 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:39.376652 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:39.376940 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:39.378196 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:39.380694 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:39.396495 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:39.396584 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:39.427464 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:05:39.438251 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:05:39.438802 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:05:39.464785 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:05:39.574707 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (310.106s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:39.640813 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:05:39.669571 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:39.670814 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:39.672029 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:39.673480 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:05:39.678483 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:39.688231 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:39.763327 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:39.765712 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:39.769528 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (310.297s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:39.770284 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240503 14:05:39.776157 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:39.777103 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:39.779547 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:39.984607 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:40.065306 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:40.065448 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:40.069383 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:40.076787 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:40.079761 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:40.089259 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:05:40.153669 16898 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (310.684s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:40.157454 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:40.160738 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:40.161136 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:40.162840 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:40.172736 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:40.210845 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:05:40.217012 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (310.744s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:40.384413 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:40.463984 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:40.467154 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:40.467633 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:40.467906 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:40.482002 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:40.557056 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:40.558245 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:40.559924 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:40.560549 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:40.567179 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:40.744194 18697 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (311.275s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:40.744918 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240503 14:05:40.762327 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (311.290s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:40.850807 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:40.853367 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:40.855669 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:40.857514 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:40.859407 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:40.944482 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:40.945060 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:40.948555 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:40.955535 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:40.955904 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:40.990660 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:41.090730 17298 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:05:41.090845 17298 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:05:41.091358 17298 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
W20240503 14:05:41.091636 12366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 14:05:41.091895 12036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:05:41.092016 12036 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:05:41.092084 12036 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:05:41.092315 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:41.092420 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:41.092473 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:41.092581 17298 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:05:41.199743 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:05:41.236848 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:41.237577 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:41.240005 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:41.249397 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:41.268450 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:41.308817 18844 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (311.836s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:41.310369 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (311.841s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:41.329335 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:05:41.340276 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:41.341619 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:41.342844 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:41.344738 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:41.354455 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:41.448293 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:41.487679 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:05:41.533574 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:05:41.636087 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:41.637658 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:41.638558 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:41.640224 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:41.648010 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:41.674559 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:05:41.686272 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:05:41.725085 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:05:41.728304 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:41.731796 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:41.733530 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:41.733949 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:41.740216 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:41.741138 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:05:41.742038 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:05:41.751840 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:05:41.772962 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:05:41.798202 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:05:41.865700 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:05:41.867524 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (312.395s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:41.881354 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:05:41.923275 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (312.454s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:42.027608 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:42.028053 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:42.036454 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:42.046700 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:42.057116 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:42.058622 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:42.100297 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:05:42.130518 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:42.133260 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:42.134008 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:42.136232 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:42.136519 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:42.155419 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:05:42.159086 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:05:42.194320 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 14:05:42.202639 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:05:42.223668 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:05:42.228878 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:05:42.383879 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (312.911s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:42.384629 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240503 14:05:42.431044 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:42.432694 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:42.440336 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:42.443608 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:42.444162 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:42.509502 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (313.040s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:42.529192 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:42.531457 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:42.531529 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:42.533002 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:42.542460 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:42.633589 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:05:42.645588 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:42.807027 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:05:42.819366 16396 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:05:42.819502 16396 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:05:42.819903 16396 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:05:42.821453 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:42.822832 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:42.822906 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:42.823252 16396 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:05:42.828867 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:42.829145 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:42.830590 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:42.831533 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:42.847406 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:42.887487 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (313.415s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:42.932713 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:42.933560 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:42.934072 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:42.934558 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:42.934890 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:43.022013 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (313.553s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:43.139847 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:43.222887 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:43.226284 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:43.226627 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:43.228240 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:43.228570 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:43.282580 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (313.810s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:43.318414 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:43.321128 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:43.321918 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:43.327510 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:43.328589 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:43.522046 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (314.053s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:43.522989 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240503 14:05:43.606719 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:43.610291 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:43.613126 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:43.616318 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:43.621531 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:43.629758 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:43.689457 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:43.712687 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:43.718811 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:43.719394 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:43.720718 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:43.722388 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:43.824245 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (314.352s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:43.960781 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:05:44.009093 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:05:44.010170 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:05:44.016971 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:44.017061 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:44.019598 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:44.019989 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:44.020514 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:44.073329 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:05:44.105677 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (314.636s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:44.120452 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:44.122347 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:44.123097 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:05:44.125784 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:44.125936 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:44.126137 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:44.127525 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:44.152153 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:05:44.181538 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:05:44.234048 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:44.249246 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:44.260949 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:44.302064 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:05:44.341455 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:44.364500 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:05:44.423370 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:44.424388 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:44.427134 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:44.429060 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:44.429553 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:44.438488 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (314.966s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:44.445567 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 14:05:44.526783 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:44.528749 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:44.529682 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:44.538470 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:44.550026 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:44.579830 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:44.676343 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:05:44.687654 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:44.724431 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (315.255s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:44.743413 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:05:44.796972 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:05:44.823927 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:44.828343 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:44.829100 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:44.831614 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:44.845626 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:44.900624 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (315.428s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:44.902411 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240503 14:05:44.922962 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:44.925963 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:44.926522 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:44.926703 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:44.935902 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:45.055586 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:05:45.220338 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:45.220603 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:45.221560 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:45.222059 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:45.231767 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:45.292626 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:45.303856 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (315.835s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:45.305922 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (315.833s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:45.314201 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:45.315197 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:45.315609 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:45.317229 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:45.318723 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:45.355926 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:05:45.611050 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:45.612114 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:45.613667 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:45.616184 11882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240503 14:05:45.616984 11882 catalog_manager.cc:4573] TS e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet faa0a6ca00d442eaaf9fca7fc1259ec7: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:45.621915 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:45.624984 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:45.707382 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:45.707489 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:45.709949 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:45.719151 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:45.725919 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:45.728426 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:45.814378 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (316.345s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:45.874272 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (316.402s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:46.003968 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:46.008433 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:46.010283 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:46.014389 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:46.015363 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:46.047983 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:05:46.104984 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:46.107197 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:46.107605 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:46.119125 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:46.120245 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:46.159754 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240503 14:05:46.219740 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (316.751s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:46.220942 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 14:05:46.246500 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:46.256460 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:46.355301 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:05:46.404105 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:46.406678 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:46.407442 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:46.407696 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:46.415637 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (316.943s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:46.416532 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:46.495409 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:05:46.506888 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:46.507655 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:46.510885 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:46.512537 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:46.516425 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:05:46.519750 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:46.530691 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:46.534920 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:46.644444 11881 catalog_manager.cc:4573] TS 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet df13ec098dc548afac8c67195d25e7e8: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:46.671586 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:46.728677 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:05:46.728912 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (317.260s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:46.746012 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:46.788928 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 14:05:46.790185 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:46.794750 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:05:46.805768 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:46.811198 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:46.813401 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:46.823670 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:46.824388 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:46.832435 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:05:46.915712 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:46.917392 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:46.917855 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:46.918668 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:46.919540 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:46.944242 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (317.472s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:46.948218 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:46.961949 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:05:46.985875 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:47.036309 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:05:47.074489 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:05:47.123560 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:47.219892 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:47.223310 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:47.223495 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:47.225219 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:47.234045 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:47.272495 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (317.803s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:47.326143 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:47.326416 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:47.329488 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:47.335592 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:47.343914 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:47.361227 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:05:47.368021 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:05:47.528671 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (318.056s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:47.529575 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 14:05:47.542400 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:47.589040 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 14:05:47.621245 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:47.625211 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:47.631655 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:47.633536 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:47.636655 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:47.722455 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:47.723619 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:47.724751 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:47.725095 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:47.726361 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:47.752727 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:05:47.806080 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (318.337s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:47.894945 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:05:48.021785 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:48.023430 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:48.023830 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:48.031430 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:48.048054 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:48.073711 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:48.111016 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (318.638s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:48.128368 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:48.131083 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:48.133013 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:48.141817 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:48.144425 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:48.300776 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (318.832s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:48.428023 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:48.430183 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:48.433601 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:48.435369 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:48.449141 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:48.530570 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:48.531654 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:48.535619 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:48.545264 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:48.547173 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:48.574565 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (319.102s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:48.584707 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:48.600795 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:05:48.717782 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:05:48.751835 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (319.283s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:48.752847 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240503 14:05:48.829545 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:48.832661 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:48.835999 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:48.838234 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:48.840247 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:48.879033 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:05:48.897696 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:05:48.923558 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:48.925081 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:48.931373 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:48.931879 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:48.940979 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:05:48.941924 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:49.036325 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (319.564s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:49.056823 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:05:49.078303 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:05:49.155972 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:05:49.160573 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:49.183724 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:05:49.212810 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:05:49.228765 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:49.229600 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:49.236126 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:49.236214 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:49.245456 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:49.334760 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:49.335050 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:49.335505 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:49.339767 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:49.347208 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:49.366401 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (319.897s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:49.368827 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:05:49.506238 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:05:49.506811 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:05:49.513125 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:05:49.552539 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:05:49.560801 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:05:49.589612 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (320.117s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:49.614670 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:49.632148 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:49.633085 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:49.635134 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:49.638101 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:49.638444 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:49.674844 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:05:49.726718 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:49.727459 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:49.728902 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:49.735684 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:49.747220 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:49.803376 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (320.334s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:49.809914 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:05:49.966728 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (320.494s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:49.967547 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240503 14:05:50.028146 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:50.032640 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:50.033454 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:50.033713 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:50.034725 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:50.133373 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:50.136265 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:50.138979 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:50.148258 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:50.149091 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:50.164382 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:05:50.166483 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:50.189296 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (320.720s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:50.250216 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:05:50.443442 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:50.450042 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:50.453474 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:50.462256 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:50.471374 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:50.478832 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:05:50.487437 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (321.015s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:50.549839 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:50.551901 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:50.552595 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:50.554391 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:50.567078 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:50.595561 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:50.807292 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (321.338s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:50.807489 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:05:50.847905 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:50.848675 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:50.851099 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:50.851393 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:50.855011 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:50.945003 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:50.947465 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (321.475s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:50.950371 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:50.951884 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:50.952375 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:50.952924 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:51.138926 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:51.188755 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:05:51.225991 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 14:05:51.228329 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (321.759s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:51.229184 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 14:05:51.238549 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:05:51.239778 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:51.239825 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:51.245450 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:51.247334 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:51.261616 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:51.329643 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (321.857s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:51.331959 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:51.335811 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:51.338765 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:51.339141 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:51.340401 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:51.346638 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:05:51.528144 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:05:51.628255 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:05:51.629356 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:05:51.630072 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:51.632835 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:51.636479 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:51.637261 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:51.639402 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:51.640182 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:05:51.672349 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:05:51.680074 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:05:51.697432 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:05:51.728154 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (322.259s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:51.728374 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:51.729619 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:51.730777 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:51.731653 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:51.749796 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:51.793928 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 14:05:51.794438 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:05:51.857365 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:05:51.888811 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (322.416s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:51.926262 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:05:51.975466 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:05:52.023734 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:52.030032 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:52.033037 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:52.033799 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:52.046299 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:52.047645 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:05:52.112740 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (322.643s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:52.132159 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:52.134629 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:52.134953 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:52.135571 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:52.141701 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:52.170814 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:52.248937 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:05:52.371590 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (322.899s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:52.372522 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240503 14:05:52.430370 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:52.430807 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:52.431054 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:52.435740 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:52.447429 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:52.524024 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:52.530014 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:52.531388 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:52.534044 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:52.541133 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:52.541711 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:05:52.545439 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:05:52.567988 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:52.632797 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (323.164s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:52.700700 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:05:52.724057 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 14:05:52.731477 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:05:52.828382 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:52.828630 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:52.828666 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:52.829485 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:52.837296 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:52.914430 18957 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (323.442s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:52.923588 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:52.924160 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:52.925653 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:52.927881 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:52.927917 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:53.031095 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:53.032390 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:05:53.165772 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (323.697s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:53.218382 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:53.218636 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:53.219230 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:53.223400 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:53.239979 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:53.313642 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:53.316512 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:53.317634 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:53.326512 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:53.336786 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:53.436384 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (323.964s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:53.442467 16396 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:05:53.442588 16396 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:05:53.442972 16396 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:05:53.443897 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:53.444429 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:53.444504 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:53.444615 16396 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:05:53.476033 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:53.554441 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (324.085s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:53.555392 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 14:05:53.570370 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:05:53.571738 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:53.611971 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:53.612591 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:53.614796 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:53.617453 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:53.619025 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:53.650338 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:05:53.705765 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:53.706061 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:53.708109 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:53.715301 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:53.716035 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:53.799916 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:05:53.838228 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (324.366s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:53.938957 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:53.946646 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (324.477s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:54.007467 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:54.008064 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:54.008158 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:54.009722 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:54.015069 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:54.080091 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:05:54.084031 17298 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:05:54.084144 17298 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:05:54.084594 17298 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:05:54.085098 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:05:54.085222 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:05:54.085290 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:05:54.086072 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:05:54.086141 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:05:54.086187 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:54.086321 17298 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:05:54.096719 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:54.096781 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:54.099179 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:54.099905 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:54.100351 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:54.110751 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:54.128108 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:05:54.167641 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:05:54.274435 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:54.300289 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (324.828s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:54.364567 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:05:54.388085 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:54.391403 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:54.393563 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:54.394138 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:54.399926 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:54.400506 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (324.931s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:54.421156 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:05:54.446077 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:05:54.473731 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:05:54.490402 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:54.491292 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:54.502529 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:54.506495 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:54.509092 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:54.526834 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:05:54.532745 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:05:54.538643 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:05:54.539171 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:05:54.599087 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:54.791966 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:54.792757 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:54.795189 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:54.795828 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:54.800311 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:54.878844 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (325.410s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:54.887439 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:54.899358 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:54.900538 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:54.901188 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:54.902611 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:54.904084 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (325.431s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:54.911469 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 14:05:55.051755 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:55.193269 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:55.194691 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:55.196134 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:55.203639 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:55.206135 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:05:55.213608 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:55.216197 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:55.230218 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:05:55.271999 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (325.803s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:55.278687 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:05:55.293134 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:55.295840 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:55.297646 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:55.307178 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:55.315600 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:55.431123 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (325.958s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:55.587800 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:55.588344 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:55.589910 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:55.592505 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:55.595360 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:55.681022 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:55.682847 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:55.685444 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:55.686430 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:55.691211 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:55.702692 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:55.878372 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (326.409s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:55.879315 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240503 14:05:55.983652 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:55.984910 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:55.986790 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:55.989936 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:55.992507 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (326.520s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:56.000216 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:56.028560 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:05:56.086655 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:56.087857 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:56.088451 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:56.089988 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:56.091408 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:56.131850 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:56.134034 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:05:56.180931 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:05:56.195945 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:05:56.234087 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 14:05:56.242573 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:56.287364 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (326.818s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:56.385666 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:56.387871 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:56.389828 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:56.390406 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:56.398967 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:56.443897 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:56.484113 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:56.484509 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:56.486426 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:56.497392 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:56.499884 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:56.598199 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (327.125s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:56.704397 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:56.708703 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (327.239s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:56.719246 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:56.783594 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:56.785816 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:56.786288 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:56.787377 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:56.792517 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:56.841846 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:56.877169 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:56.877168 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:56.878191 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:56.881079 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:56.889912 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:56.909476 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:05:56.953874 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:05:56.963784 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 14:05:56.964354 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:05:56.998641 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:57.004555 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (327.532s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:57.012111 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:05:57.056021 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:05:57.160615 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:57.180598 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:57.183210 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:57.183887 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:57.187129 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:57.189638 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:57.191962 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:05:57.204522 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (327.735s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:57.275624 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:57.278261 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:57.280535 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:57.283939 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:57.285317 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:57.354992 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:05:57.368953 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:57.568151 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:57.568493 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:57.568887 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (328.096s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:57.568894 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:57.569921 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240503 14:05:57.572944 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:57.573056 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:57.573675 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:57.612133 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:57.659202 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:57.664052 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:57.664880 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:57.666095 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:57.679476 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:57.693918 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:57.745878 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (328.277s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:57.818703 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 14:05:57.822010 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:05:57.902468 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:05:57.958066 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:57.961848 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:57.962277 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:57.963997 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:57.973527 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:58.054400 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:58.056136 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:58.057142 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:58.057510 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:58.068353 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:58.108698 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:58.131808 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (328.659s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:58.349624 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:58.350623 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:58.350857 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:58.352996 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:58.362865 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (328.894s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:58.365288 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240503 14:05:58.367395 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:58.459123 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:58.461107 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:58.462572 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:58.463402 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:58.463812 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:58.647768 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:58.672748 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:05:58.686098 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (329.213s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:58.708730 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:05:58.752220 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:58.753247 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:58.755009 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:58.756889 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:58.757066 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:58.770522 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:05:58.797022 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:05:58.844720 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:58.846273 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:58.846922 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:58.846959 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:58.849421 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:58.943100 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (329.474s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:58.945428 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:05:58.990509 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:05:59.115187 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:05:59.138576 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:59.138983 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:59.139444 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:05:59.139760 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:05:59.142426 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:59.151392 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:59.227306 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (329.755s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:59.235707 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:59.238312 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:59.238592 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:59.238669 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:05:59.248211 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:05:59.249552 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:59.358362 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:05:59.359820 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:05:59.404839 21694 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:05:59.404961 21694 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:05:59.405457 21694 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:05:59.406034 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:05:59.406181 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:05:59.406260 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:05:59.406360 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:05:59.406601 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:05:59.406666 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:05:59.406906 21694 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:05:59.407680 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:05:59.435202 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:05:59.485054 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (330.016s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:59.533463 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:59.536581 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:59.538436 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:59.547828 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:59.557523 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:59.591408 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:05:59.617355 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:05:59.617959 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:05:59.626672 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:59.629047 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:59.630479 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:59.640357 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:59.641721 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:05:59.654428 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 14:05:59.669499 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:05:59.849203 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (330.376s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:05:59.924597 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:05:59.925429 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:05:59.929536 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:05:59.929771 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:59.932072 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:05:59.986668 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (330.517s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:05:59.997805 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:06:00.015504 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:00.015775 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:00.019712 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:00.026705 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:00.033666 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:00.091331 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:06:00.170258 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:00.218233 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:06:00.275978 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:06:00.316740 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (330.844s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:00.317788 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240503 14:06:00.323401 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:00.326114 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:00.328595 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:00.329859 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:00.338217 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:00.425251 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:00.428660 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:00.428772 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:00.430018 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:00.430044 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:00.468290 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:06:00.557183 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:00.586045 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (331.117s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:00.733606 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:00.734802 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:00.735198 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:00.735558 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:00.746937 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:00.841102 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:00.845366 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:00.845574 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:00.846053 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:00.847410 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:00.930708 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (331.458s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:01.014430 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:01.020246 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (331.551s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:01.021092 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240503 14:06:01.145397 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:01.145643 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:06:01.146253 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:01.149969 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:01.152376 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:01.167279 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:01.209540 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:06:01.256489 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:06:01.263674 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:01.263928 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:01.264145 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:01.264608 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:01.277184 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:01.421957 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:06:01.425169 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (331.956s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:01.450754 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:06:01.469696 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (331.997s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:01.507575 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:01.512887 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:06:01.518949 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 22 similar messages]
I20240503 14:06:01.561667 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:01.565079 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:01.566005 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:01.566515 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:01.576406 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:01.666141 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:06:01.667935 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:01.669123 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:01.669901 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:01.677588 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:01.690821 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:01.719694 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:06:01.859467 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:06:01.885399 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:06:01.890682 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (332.422s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:01.918596 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:06:01.919337 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:06:01.948189 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (332.476s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:01.976071 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:01.976261 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:01.978247 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:01.981863 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:01.990341 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:02.000533 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 14:06:02.001171 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:02.039853 11882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 14:06:02.040923 11882 catalog_manager.cc:4573] TS e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet faa0a6ca00d442eaaf9fca7fc1259ec7: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:06:02.076691 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:02.076968 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:02.080237 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:02.088876 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:02.105297 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:02.106570 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:06:02.174791 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:06:02.200974 22902 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:06:02.201120 22902 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:06:02.201550 22902 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:06:02.202395 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:06:02.202502 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:06:02.202576 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:06:02.202680 22902 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:06:02.361855 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (332.889s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:02.380985 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:06:02.389062 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:02.390619 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:02.396050 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:02.396418 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:02.410492 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:02.477854 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (333.009s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:02.498931 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:02.500973 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:02.504155 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:02.504699 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:02.505373 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:02.537598 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:02.577749 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:06:02.643219 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:06:02.657608 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:06:02.756677 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (333.284s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:02.757654 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240503 14:06:02.806181 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:02.807808 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:02.808681 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:02.808885 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:02.809290 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:02.820367 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:06:02.828742 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:06:02.864547 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (333.395s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:02.906692 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:02.909991 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:06:02.911854 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 14:06:02.914654 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:02.915597 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:02.918186 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:02.925262 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:03.072510 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:03.218844 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:03.219892 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:03.220108 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:03.223207 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:03.231424 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:03.265448 11881 catalog_manager.cc:4573] TS 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c42fee9fadbb4601a7fe32eb6a04e767: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:06:03.309057 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (333.836s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:03.327891 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:03.328346 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:03.329219 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:03.333258 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:03.351102 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:03.410329 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (333.941s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:03.411324 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 14:06:03.608407 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:06:03.610430 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:03.634095 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:03.634142 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:03.635423 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:03.640331 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:03.650280 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:03.662742 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:03.694214 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:03.745550 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:03.745553 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:03.748245 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:03.749003 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:03.755066 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:03.872325 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (334.400s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:03.879644 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:06:03.907063 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (334.438s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:03.930724 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:04.064839 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:04.066443 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:04.067095 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:04.069185 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:04.072050 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:06:04.076714 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:06:04.087340 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:04.168859 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:04.170641 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:04.171792 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:04.175588 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:04.182432 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:04.222275 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:04.240773 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:06:04.297847 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:06:04.379783 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (334.911s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:04.417729 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:06:04.430583 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:06:04.439402 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:06:04.450140 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:06:04.474514 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:04.475955 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:04.479640 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:04.480958 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:04.491813 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:04.497385 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (335.025s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:04.575570 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:04.577162 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:04.579334 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:04.579524 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:04.584692 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:04.634321 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:04.776325 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (335.307s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:04.806401 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:04.876646 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:04.880116 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:04.881491 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:04.883658 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:04.885136 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:04.906716 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:06:04.920347 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:06:04.920996 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:06:04.942351 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (335.470s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:04.979607 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:04.984408 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:04.986538 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:04.987164 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:04.991464 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:05.237507 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:06:05.244340 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:05.290508 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:05.291509 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (335.822s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:05.293419 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:05.295934 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:05.296344 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:05.299189 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:05.331151 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:06:05.393080 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:05.394018 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:05.394044 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:05.397064 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:05.399554 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:05.415916 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:05.484529 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (336.012s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:05.485570 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 14:06:05.672179 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:06:05.696877 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:05.702879 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:05.703395 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:05.703935 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:05.715117 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:05.717470 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:05.805852 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:05.806663 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:05.808431 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:05.808923 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:05.819898 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:05.851688 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (336.382s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:05.852707 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240503 14:06:05.871699 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (336.399s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:06.111268 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:06.111505 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:06.111706 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:06.112109 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:06.123497 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:06.125696 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:06.199692 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:06:06.206871 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:06.207512 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:06.214958 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:06.219748 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:06.240222 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:06.251631 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:06.271180 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (336.798s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:06.336400 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:06.358321 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:06.395790 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (336.927s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:06.420411 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:06.430531 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:06:06.498191 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:06.521064 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:06.523195 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:06.527433 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:06.532909 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:06.540531 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:06.570617 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:06.625383 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:06.626010 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:06.627621 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:06.629211 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:06.631403 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:06.682549 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:06:06.773530 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:06:06.786867 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:06.790450 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (337.318s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:06.793402 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 14:06:06.893715 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (337.425s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:06.904484 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:06:06.932880 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:06.933162 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:06.933162 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:06.940529 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:06.943423 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:07.019276 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:06:07.028439 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:07.032780 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:07.036387 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:07.044294 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:07.046487 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:07.057487 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:07.081884 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:06:07.100697 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:07.187008 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:07.213181 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (337.741s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:07.223999 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:07.345472 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:07.345944 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:07.346608 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:07.347007 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:07.347312 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (337.878s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:07.347918 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:07.444161 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:07.449451 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:07.450081 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:07.468349 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:07.471136 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:07.471669 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:06:07.610993 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:07.698879 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:07.739185 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:07.756646 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:07.757793 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:07.758468 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:07.758690 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:07.759265 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:07.823755 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (338.351s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:07.824779 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 14:06:07.828915 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:06:07.859366 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:07.860350 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:07.860558 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:07.863204 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:07.863726 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:07.869446 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (338.400s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:07.997985 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:06:08.044279 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 14:06:08.110337 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:06:08.112847 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:08.168211 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:08.169025 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:08.170397 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:08.172094 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:08.177744 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:08.273887 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:08.279210 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:08.280618 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:08.289575 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:08.289582 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:08.351742 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (338.883s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:08.352653 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240503 14:06:08.392807 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (338.920s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:08.400182 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:08.577970 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:08.580736 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:08.582825 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:08.584609 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:08.600286 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:08.602481 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:06:08.650873 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:08.690863 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:08.691370 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:08.694506 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:08.702590 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:08.702775 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:08.792160 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:08.846969 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:06:08.885804 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (339.417s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:08.930521 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (339.458s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:08.990119 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:08.995628 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:08.995898 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:08.995982 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:08.997341 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:08.999423 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:09.085436 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:09.091452 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:06:09.103732 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:09.106073 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:09.106590 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:09.109495 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:09.120114 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:09.158998 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:09.162575 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:06:09.169481 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:09.220916 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:09.236415 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:09.336531 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (339.867s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:09.383890 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:09.410565 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:09.411782 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:09.412568 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:09.413601 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:09.423346 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:09.430856 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (339.958s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:09.431679 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:06:09.516090 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:09.519073 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:09.519150 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:09.519680 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:09.520305 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:09.661164 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:09.684082 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:09.748278 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:09.811869 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:09.814411 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:09.816776 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:06:09.818801 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:09.821173 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:09.827503 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:09.879669 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (340.410s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:09.914605 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:09.917480 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:09.917626 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:09.930531 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:09.939438 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:09.956951 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (340.484s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:09.990497 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 14:06:10.032433 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:06:10.107280 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:10.226186 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:10.226307 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:10.229271 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:06:10.229982 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:10.236477 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:10.239701 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:10.256634 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (340.787s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:10.326808 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:10.336169 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:10.338164 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:10.344910 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:10.345602 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:10.355222 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:10.457876 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (340.985s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:10.458683 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 14:06:10.572880 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:06:10.638243 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:10.641534 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:10.644517 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:10.647588 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:10.658331 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:10.703831 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:10.751513 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:10.751613 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:10.760248 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:10.760463 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:10.763705 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:10.782908 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (341.314s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:10.784257 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 14:06:10.856365 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:10.859659 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (341.387s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:10.977038 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:11.060340 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:11.064987 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:11.067389 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:11.068605 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:11.084009 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:11.158411 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (341.689s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:11.170502 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:11.170703 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:11.178754 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:11.179545 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:11.188786 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:11.261189 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:11.301144 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (341.828s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:11.345862 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:06:11.407111 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:11.486485 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:11.486991 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:11.487015 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:11.487390 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:11.502672 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:11.569424 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (342.100s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:11.585994 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:11.586360 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:11.588399 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:11.590932 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:11.591351 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:11.593482 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:11.607105 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:11.644142 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:06:11.710153 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:06:11.715159 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:06:11.771911 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:11.794631 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 14:06:11.804852 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:06:11.817991 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:11.826956 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (342.354s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:11.895581 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:11.895653 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:11.896075 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:11.896982 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:11.898480 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:11.911420 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:11.913421 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:12.000517 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:12.001241 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:12.001987 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:12.003922 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:12.006997 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:12.007402 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:12.038210 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 111 similar messages]
I20240503 14:06:12.149310 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (342.680s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:12.223284 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:12.250459 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (342.778s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:12.299839 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:12.301043 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:12.307544 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:12.308846 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:12.324687 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:12.362432 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:12.404637 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:12.405941 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:12.409718 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:12.411684 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:12.416543 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:12.420212 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:12.473398 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:06:12.646755 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:12.699797 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (343.227s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:12.700978 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240503 14:06:12.715983 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:12.718645 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:12.718911 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:12.719882 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:12.723421 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:12.737560 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (343.268s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:12.751598 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:06:12.820678 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:12.826269 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:12.827890 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:12.828567 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:12.828980 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:12.835319 24481 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:06:12.835423 24481 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:06:12.835886 24481 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:06:12.836746 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:06:12.836874 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:06:12.836939 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:06:12.837203 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:06:12.837664 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:06:12.837736 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:06:12.838495 24481 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:06:12.906657 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:06:12.934302 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:13.045661 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:13.081825 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (343.609s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:13.125079 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:13.129776 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:13.142966 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (343.674s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:13.149533 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:13.155349 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240503 14:06:13.159844 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:13.163422 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:13.273002 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:13.274678 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:13.274948 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:13.275801 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:13.287842 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:13.336997 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 14:06:13.341133 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:06:13.455514 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:13.539122 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:13.576825 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:13.577394 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:13.584795 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:13.586232 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:13.587674 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:13.599714 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (344.127s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:13.603098 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (344.134s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:13.624241 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:13.680400 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:13.680648 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:13.684362 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:13.691367 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:13.694113 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:13.927527 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:06:13.957578 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:13.990882 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:13.992820 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:13.993002 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:13.995221 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:14.001319 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:14.037994 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:14.065182 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (344.592s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:14.098052 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:14.098469 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:14.101439 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:14.102823 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:14.109606 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:14.112763 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:06:14.145494 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:14.201314 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (344.732s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:14.219038 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:06:14.311264 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:06:14.334059 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:06:14.374229 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:14.395560 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:14.396709 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:14.398602 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:14.398697 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:14.401557 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:14.422585 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:06:14.489588 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:14.500440 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:14.503433 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:14.505221 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:14.507061 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:14.517527 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:14.532195 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (345.060s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:14.545462 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:06:14.564504 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:14.723949 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (345.255s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:14.803663 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:14.806778 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:14.815341 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:14.822435 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:14.831600 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:14.836073 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:14.896131 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:14.922474 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:14.923748 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:14.925673 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:14.931416 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:14.946928 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:15.011572 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:06:15.018121 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:15.041692 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (345.569s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:15.042786 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 14:06:15.068976 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:15.137542 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:15.141808 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (345.673s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:15.196367 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:15.231032 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:15.231577 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:15.232582 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:15.235055 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:15.257143 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:15.343992 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:15.344513 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:15.346935 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:15.348811 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:15.349051 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:15.359644 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:06:15.482434 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (346.010s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:15.489269 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:06:15.506856 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:15.644260 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:15.647543 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:15.648164 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:15.649617 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:15.652874 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:15.702555 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:15.752650 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:15.753399 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:15.754782 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:15.757916 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:15.758605 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:15.766049 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (346.297s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:15.769632 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240503 14:06:16.057456 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:16.059384 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:16.060911 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:16.071063 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:16.072926 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:16.098021 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (346.625s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:16.119587 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:16.127425 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:06:16.157912 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:16.161828 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:16.162549 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:16.166165 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:16.176539 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:16.217868 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:16.248082 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (346.779s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:16.469128 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:16.470415 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:16.472033 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:16.472110 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:16.485153 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:16.493852 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (347.021s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:16.537158 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:06:16.570509 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:16.571887 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:16.574357 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:16.575404 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:16.577451 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:16.645756 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:16.658941 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:06:16.725196 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (347.256s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:16.784763 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:06:16.805037 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:06:16.819897 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:06:16.835906 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:06:16.869432 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:16.871430 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:16.876027 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:16.876603 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:16.876735 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:16.888651 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:16.939085 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (347.466s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:16.980253 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:16.980656 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:16.983276 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:16.984053 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:16.995975 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:17.028146 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240503 14:06:17.063973 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:06:17.091105 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 14:06:17.095314 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:17.145196 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:06:17.249264 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:17.282893 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:17.285516 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:17.285879 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:17.291906 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:17.301757 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:17.338143 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (347.869s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:17.354152 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:17.385443 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:17.386250 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:17.386991 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:17.387235 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:17.395259 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:17.460500 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (347.988s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:17.461496 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 14:06:17.502156 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:06:17.567031 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:06:17.624869 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:06:17.672644 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:17.686650 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:17.690752 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:17.691663 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:17.699664 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:17.700897 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:17.701275 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:17.755194 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (348.286s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:17.789203 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:17.790913 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:17.799365 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:17.800213 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:17.801180 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:17.883157 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:06:17.896683 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:06:18.073304 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (348.601s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:18.102492 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:18.103874 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:18.106249 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:18.108160 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:18.108595 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:18.201789 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:18.202471 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:18.202559 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:18.204483 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (348.735s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:18.211390 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:18.213186 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240503 14:06:18.219722 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:18.257793 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:18.414678 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 14:06:18.416990 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:06:18.506806 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:18.507860 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:18.510668 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:06:18.512089 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:18.514529 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:18.519438 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:18.521755 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:06:18.590093 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (349.121s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:18.612337 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:18.613488 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:18.617177 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:18.617283 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:18.620209 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:18.621418 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (349.149s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:18.668135 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:06:18.875293 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:18.922619 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:18.923175 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:18.924778 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:18.925341 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:18.931391 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:19.008953 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:06:19.015302 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:06:19.026654 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:19.027417 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:19.027484 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:19.035392 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:19.035666 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:19.114724 14821 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (349.646s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:19.174665 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (349.702s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:19.252296 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:06:19.337726 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:19.341147 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:19.342646 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:19.347365 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:19.348032 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:19.368902 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:06:19.437703 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:19.440320 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:06:19.448418 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:19.455826 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:19.458025 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:19.468154 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:19.476992 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:06:19.478555 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:19.505954 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:06:19.512203 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:06:19.601344 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:06:19.702358 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (350.233s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:19.733804 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (350.261s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:19.753989 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:19.760006 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:19.762483 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:19.762902 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:19.763525 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:19.789686 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:06:19.865921 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:19.866750 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:19.867677 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:19.869792 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:19.874544 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:19.882284 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:06:19.939864 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:20.041771 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:06:20.062793 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240503 14:06:20.063421 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:06:20.121712 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (350.653s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:20.174177 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:20.174978 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:20.175806 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:20.176288 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:20.183414 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:20.214449 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (350.742s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:20.215636 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 14:06:20.248162 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:06:20.273535 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:06:20.276306 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:06:20.281293 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:20.281472 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:20.284981 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:20.292025 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:20.302150 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:20.404688 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:06:20.492429 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:20.543022 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (351.074s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:20.544121 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240503 14:06:20.588732 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:20.588842 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:20.589977 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:20.591318 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:20.601130 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:20.689173 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (351.216s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:20.699960 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:20.701328 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:20.705178 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:20.706265 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:20.708113 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:20.972751 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:06:20.985450 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:06:20.987540 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:21.007380 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:21.010655 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:21.012513 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:21.013839 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:21.015062 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:21.082325 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:06:21.103833 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (351.635s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:21.113876 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:21.116308 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:21.117311 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:21.118983 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:21.119390 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:21.242839 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (351.770s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:21.426715 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:21.429060 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:21.435992 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:21.438083 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:21.439383 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:21.509191 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:06:21.537787 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:21.537882 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:21.539048 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:21.540928 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:21.548771 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:21.568326 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:06:21.569343 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:21.587460 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:06:21.704676 21363 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (352.236s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:21.709910 22902 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:06:21.710022 22902 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:06:21.710523 22902 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:06:21.711424 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:06:21.711578 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:06:21.711652 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:06:21.711776 22902 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:06:21.716895 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:06:21.809078 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:06:21.831434 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (352.359s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:21.842906 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:21.844185 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:21.845348 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:21.848007 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:21.848718 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:21.915089 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:06:21.949482 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:21.950181 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:21.951278 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:21.952543 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:21.953089 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:22.097694 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (352.628s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:22.098464 12235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 14:06:22.099619 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:06:22.118966 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 14:06:22.133409 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:06:22.138741 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:06:22.187270 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:22.206763 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (352.734s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:22.250770 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:22.251143 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:22.256078 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:22.257313 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:22.266726 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:22.360589 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:22.365693 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:22.366468 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:22.368062 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:22.381526 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:06:22.381829 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:22.396346 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:06:22.497990 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (353.029s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:22.541260 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:06:22.673440 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:22.673889 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:22.674499 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:22.675580 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:22.675981 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:22.677311 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:22.684865 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (353.212s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:22.686762 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 14:06:22.738675 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:06:22.742381 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:06:22.782454 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:22.785256 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:22.786942 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:22.791575 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:22.795145 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:22.872730 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:06:22.917287 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:06:23.030184 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (353.561s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:23.031387 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 14:06:23.062695 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:06:23.086859 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:23.088145 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:23.092725 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:23.093098 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:23.093387 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:23.104758 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:23.195868 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:23.196272 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:23.196332 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:23.212782 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:23.219832 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:23.245539 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (353.773s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:23.395560 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:06:23.408771 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:06:23.448714 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:06:23.509786 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:23.513437 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:23.514297 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:23.525377 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:23.526808 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:23.536016 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:06:23.542313 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (354.073s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:23.619838 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:23.620952 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:23.621006 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:23.622149 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:23.631848 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:23.676846 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:23.756453 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (354.284s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:23.762250 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 14:06:23.927071 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:23.928537 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:23.928625 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:23.936014 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:23.942555 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:24.016016 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:06:24.032742 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:24.036011 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:24.037672 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:24.039808 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:24.041498 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:24.060075 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (354.591s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:24.088034 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:24.153880 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:06:24.183490 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:06:24.188912 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:06:24.209640 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:06:24.228644 27272 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:06:24.228824 27272 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:06:24.229465 27272 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:06:24.230026 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:06:24.230154 12035 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:06:24.230233 12035 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:06:24.230662 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:06:24.230739 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:06:24.230816 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:06:24.230948 27272 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:06:24.291139 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:06:24.342463 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:24.348510 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:24.348668 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:24.349140 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:24.351383 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:24.367630 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (354.895s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:24.451328 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:24.453449 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:24.453673 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:24.462545 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:24.471426 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:24.530344 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:06:24.573022 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (355.104s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:24.619841 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:06:24.622665 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:06:24.663025 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:24.742239 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:06:24.763440 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:24.764333 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:24.765506 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:24.766471 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:24.771495 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:24.857640 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:24.858254 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:24.861281 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:24.861292 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:24.863772 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (355.391s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:24.871342 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:24.940133 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:06:25.152212 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:25.155869 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:25.156222 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:25.158172 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (355.689s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:25.159384 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:25.168552 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:25.170455 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:25.171659 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:06:25.181605 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:06:25.242271 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:06:25.255589 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:25.255750 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:25.263490 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:25.265427 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:25.275410 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:25.363694 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:06:25.389361 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:06:25.423194 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (355.951s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:25.424187 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 14:06:25.518767 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:06:25.553665 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:25.553881 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:25.556071 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:25.559595 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:25.561966 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:25.651120 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:25.651502 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:25.659756 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:25.661594 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:25.662432 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:25.682271 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (356.213s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:25.685376 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 14:06:25.782769 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:06:25.784914 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:25.819281 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:06:25.887939 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (356.415s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:25.897282 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:06:25.932550 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:06:25.953279 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:25.953284 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:25.958850 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:25.960714 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:25.964067 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:26.053906 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:26.058017 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:26.058357 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:26.069569 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:26.081321 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:26.232201 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (356.763s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:26.310179 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:26.359982 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:26.360234 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:26.360350 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:26.367712 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:26.368106 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:26.461560 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (356.989s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:26.481302 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:26.481950 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:26.482923 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:26.486580 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:26.487455 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:26.524103 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:06:26.648051 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:06:26.688766 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (357.220s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:26.710057 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:26.719214 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:06:26.775326 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:26.777768 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:26.778347 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:06:26.780081 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:26.786701 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:26.787326 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:26.875675 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:26.877435 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:26.878625 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:26.887722 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:26.895387 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:26.904870 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:06:26.914319 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:06:26.987704 27272 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:06:26.987825 27272 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:06:26.988235 27272 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:06:26.988723 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:06:26.988843 12035 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:06:26.988910 12035 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:06:26.989199 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:06:26.989287 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:06:26.989365 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:06:26.989497 27272 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:06:27.031184 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (357.558s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:27.066439 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:06:27.122476 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:06:27.128512 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 14:06:27.177975 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 23 similar messages]
I20240503 14:06:27.183889 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:27.185662 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:27.185817 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:27.186098 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:27.189911 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:27.255897 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:27.272169 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:06:27.275918 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (357.807s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:27.278873 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:27.278915 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:27.281713 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:27.288151 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:27.288699 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:27.415550 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (357.943s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:27.440192 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:06:27.580507 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:27.583369 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:27.583513 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:27.584240 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:27.592146 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:27.599964 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:06:27.675417 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:27.677578 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:27.678548 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:27.679761 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:27.691733 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:06:27.692544 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:27.742300 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (358.273s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:27.800088 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:06:27.831599 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (358.359s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:27.832625 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 14:06:27.860991 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:27.886189 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:06:27.964021 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:27.967569 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:27.973963 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:27.974201 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:27.976243 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:27.987907 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:06:28.062623 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:28.063529 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:28.067658 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:28.071779 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:28.082212 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:28.145251 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:06:28.149761 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:06:28.269265 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:06:28.304358 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:28.347499 27434 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (358.878s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:28.348278 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240503 14:06:28.349795 26943 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (358.877s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:28.353384 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:28.356509 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:28.360158 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:28.360306 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:28.360981 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:28.391423 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:06:28.453588 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:28.453958 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:28.455935 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:28.461444 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:28.476351 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:28.489060 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:06:28.679208 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:06:28.757191 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:28.757894 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:28.758378 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:28.759835 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:28.768492 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:28.853360 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:28.853401 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:28.855458 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:28.857209 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:28.864156 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:28.864490 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:28.921960 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (359.449s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:28.966718 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (359.497s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:29.130306 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:06:29.148799 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:29.151916 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:29.152756 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 14:06:29.152812 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:29.153211 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:29.155403 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:29.221603 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:06:29.254781 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:06:29.255826 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:06:29.283164 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:06:29.328356 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:29.381408 28776 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (359.912s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:29.385083 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:06:29.484249 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:06:29.490211 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (360.018s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:29.673909 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:06:29.708909 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:06:29.719162 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:06:29.870659 28776 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (360.398s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:29.887517 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:29.900272 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (360.431s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:30.090395 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:06:30.168773 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:06:30.179683 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:06:30.180239 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:06:30.216073 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:30.216279 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:30.216279 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:30.220394 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:30.230839 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:30.269905 28776 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (360.797s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:30.270653 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240503 14:06:30.316108 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (360.847s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:30.343595 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:06:30.415472 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:06:30.443671 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:06:30.458570 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:30.513221 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:30.514405 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:30.514539 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:30.515051 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:30.521404 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:30.612566 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:30.612639 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:30.612932 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:30.613431 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:30.617440 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:30.638459 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:06:30.747967 28776 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (361.275s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:30.782716 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (361.314s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:30.783710 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 14:06:30.799471 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:06:30.822113 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:06:30.908195 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:30.913013 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:30.915602 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:30.927282 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:30.929922 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:31.015291 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:31.015656 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:31.018611 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:31.021198 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:31.041077 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:31.048871 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:31.072690 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:31.322634 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:31.323299 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:31.323472 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:31.333295 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:31.334971 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (361.862s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:31.338201 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:31.369884 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:31.404548 29217 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (361.935s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:31.422560 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:31.424530 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:31.426368 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:31.430883 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:31.443425 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:31.548374 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:31.574720 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:06:31.660156 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:06:31.717514 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:31.727424 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:31.729561 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:31.731091 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:31.747713 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:31.751840 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:31.818351 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:06:31.829953 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:31.830032 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:31.841030 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:31.842447 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:31.844254 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:31.913312 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:31.936802 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (362.464s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:31.973068 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:32.015626 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (362.546s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:32.017424 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:06:32.126644 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:32.129297 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 14:06:32.129894 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:06:32.130100 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:32.131423 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:32.135596 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:32.139117 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:32.144358 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:06:32.202163 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 14:06:32.220417 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:32.224445 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:32.224782 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:32.225016 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:32.235551 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:32.428849 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:32.430984 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (362.958s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:32.434990 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:06:32.524998 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:32.525056 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:32.525884 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:32.525974 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:32.537480 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:32.573253 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (363.104s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:32.609022 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:32.614888 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:32.615957 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:32.617394 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:32.619597 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:32.620887 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:32.710155 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:06:32.791880 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:06:32.857785 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:06:32.911084 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:32.911569 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:32.912583 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:32.923580 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:32.934804 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:32.989109 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:06:32.994145 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (363.521s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:32.995491 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240503 14:06:33.006457 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:33.011803 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:33.012220 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:33.014999 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:33.016099 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:33.027768 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:33.054884 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:33.066159 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (363.597s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:33.097056 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:33.308815 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:33.310367 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:33.313621 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:33.313678 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:33.314445 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:33.403677 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:33.403942 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:33.404889 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:33.405241 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:33.405891 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:33.430666 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:33.440778 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:06:33.456972 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:33.525012 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (364.056s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:33.525938 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240503 14:06:33.610236 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:06:33.611318 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (364.139s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:33.661337 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:33.701540 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:33.701977 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:33.704020 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:33.704902 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:33.705355 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:33.792090 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:33.795439 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:33.800340 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:33.805599 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:33.812191 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:33.904898 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:06:33.932225 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (364.463s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:34.005667 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:34.024519 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:06:34.025097 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:34.075222 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (364.603s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:34.087404 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:34.089769 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:34.092195 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:34.105406 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:34.105780 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:34.185478 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:34.186748 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:34.197300 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:34.198180 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:34.198843 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:34.265146 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:34.408232 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (364.939s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:34.434844 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:06:34.434870 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:34.486625 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:34.488709 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:34.490407 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:34.495306 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:34.499307 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:34.556953 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:34.579994 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (365.107s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:34.584628 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:34.588490 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:34.588725 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:34.593223 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 14:06:34.593899 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:34.595968 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:34.596031 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:34.672339 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:06:34.793081 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:06:34.837759 11882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 14:06:34.838434 11882 catalog_manager.cc:4573] TS e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet faa0a6ca00d442eaaf9fca7fc1259ec7: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:06:34.838492 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:34.852355 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (365.383s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:34.879577 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:34.879981 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:34.883189 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:34.884618 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:34.885178 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:34.977212 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:34.978093 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:34.980216 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:34.980343 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:34.981089 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:34.990952 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:35.025233 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:35.081845 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (365.609s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:35.145015 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:06:35.239823 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:35.272346 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (365.803s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:35.278439 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:35.280879 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:35.281381 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:35.291499 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 14:06:35.297547 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:35.297690 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:35.367777 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:35.369809 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:35.371924 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:35.373620 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:35.377107 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:35.392596 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:06:35.487771 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:35.496408 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (366.024s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:35.497262 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 14:06:35.511595 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:35.581172 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:06:35.605074 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:06:35.669406 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:35.671849 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:35.675336 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:35.678162 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:35.692051 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:35.765476 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:35.768604 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:35.771473 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:35.779855 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:35.780185 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:35.833047 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (366.364s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:35.833928 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 14:06:35.842937 11881 catalog_manager.cc:4573] TS 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet df13ec098dc548afac8c67195d25e7e8: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:06:35.845299 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:06:35.902268 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:06:36.026724 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:06:36.066617 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:36.066929 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:36.068332 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:36.073304 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:36.074980 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:36.092121 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:36.093011 28905 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (366.620s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:36.159276 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:06:36.169281 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:36.170029 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:36.170576 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:36.171047 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:36.173004 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:36.344244 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (366.875s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:36.467466 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:36.488494 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:36.491407 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:36.506372 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:36.509928 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:36.516902 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:36.551409 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:36.597056 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:36.597272 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:36.599859 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:36.600258 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:36.600584 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:36.664628 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (367.192s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:36.789433 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:06:36.833856 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:06:36.840071 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (367.371s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:36.881675 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:36.882712 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:36.887696 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:36.887897 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:36.888231 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:36.909653 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:36.979826 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:36.981215 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:36.982920 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:36.983898 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:36.989537 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:36.997308 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:37.052551 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:37.158140 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 14:06:37.161584 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:06:37.224995 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (367.752s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:37.288008 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:37.288532 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:37.290939 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:37.299624 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:37.301728 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:37.355046 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:37.370031 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (367.901s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:37.377977 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:37.381198 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:37.383141 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:37.383316 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:37.387529 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:37.416023 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 14:06:37.416519 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:06:37.472877 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:06:37.550909 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:37.569902 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:06:37.618868 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:06:37.635972 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:37.671840 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:37.673748 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:37.678121 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:37.678347 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:37.683600 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (368.211s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:37.700167 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:37.774694 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:37.776443 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:37.777292 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:37.778019 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:37.780390 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:37.899204 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:37.957960 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (368.489s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:38.041416 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:06:38.071072 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:38.072695 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:38.074637 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:38.076929 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:38.077445 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:38.113191 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (368.641s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:38.114224 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240503 14:06:38.135290 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:06:38.136453 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:06:38.138916 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:06:38.158229 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:06:38.164575 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:38.168974 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:38.172725 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:38.175549 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:38.176067 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:38.177723 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:38.473337 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:38.473750 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:38.475417 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:38.486533 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:38.491919 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:38.530825 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (369.062s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:38.531839 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240503 14:06:38.577539 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:38.579111 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:38.581004 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:38.581054 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:38.589062 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:38.590358 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (369.118s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:38.729071 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:38.730871 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:06:38.740218 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:06:38.873276 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:38.873805 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:38.877171 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:38.877529 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:38.883378 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:38.946439 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (369.477s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:38.966243 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:38.966477 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:38.970698 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:38.973062 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:38.975873 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:39.003226 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:06:39.071087 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:06:39.072301 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:06:39.130471 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:39.175443 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (369.703s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:39.262470 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:39.269146 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:39.269269 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:39.275660 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:39.278124 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:39.296873 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:06:39.324684 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:06:39.362650 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:39.363422 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:39.363595 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:39.368194 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:39.373805 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:39.401638 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (369.932s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:39.409358 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:06:39.622221 22902 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:06:39.622342 22902 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:06:39.622762 22902 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:06:39.623119 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 14:06:39.623745 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:06:39.623993 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:06:39.624063 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:06:39.624186 22902 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:06:39.645020 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:39.660423 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:39.661351 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:39.664738 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (370.192s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:39.665982 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:39.667219 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:39.668401 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:39.761353 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:39.762941 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:39.765622 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:39.771510 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:39.777947 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:39.835836 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:06:39.923071 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:06:39.935307 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (370.466s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:40.004348 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:06:40.007936 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:06:40.055831 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:40.055936 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:40.059031 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:40.059577 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:40.075989 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (370.603s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:40.078580 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:40.098919 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:06:40.149912 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:40.150732 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:40.151507 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:40.156641 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:40.159250 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:40.159786 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:06:40.164935 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:40.286067 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:06:40.353988 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:06:40.354617 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:06:40.447582 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:40.450213 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:40.451715 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:40.454617 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:40.457135 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:40.463657 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (370.994s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:40.520471 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:40.544951 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:40.545352 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:40.546685 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:40.547084 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:40.547713 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:40.561292 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:06:40.640643 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:06:40.680425 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (371.208s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:40.680639 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:06:40.681166 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 14:06:40.690392 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:06:40.724182 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:40.837690 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:40.838630 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:40.839816 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:40.844987 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:40.849793 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:40.927781 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:40.928382 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:40.929734 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:40.939280 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:40.940502 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:41.001857 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:06:41.013947 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (371.545s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:41.015194 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240503 14:06:41.182778 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (371.710s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:41.218080 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:41.223930 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:41.224449 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:41.225248 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:41.226862 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:41.268296 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:41.316509 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:41.316841 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:41.318420 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:41.329910 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:41.330137 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:41.337292 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:06:41.396185 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (371.927s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:41.446686 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:41.545886 29302 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:06:41.546017 29302 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:06:41.546582 29302 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:06:41.547199 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:06:41.547336 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:06:41.547407 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:06:41.547598 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:06:41.547691 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:06:41.547777 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:06:41.547926 29302 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:06:41.611816 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:41.614043 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:41.616724 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:41.619068 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:41.620108 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:41.631294 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:06:41.661717 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:41.706534 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:41.709882 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:41.710876 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:41.712561 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:41.729557 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:41.790848 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (372.318s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:41.835098 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:41.955549 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:06:41.987555 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (372.518s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:42.008219 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:42.008654 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:42.009145 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:42.011461 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:42.019186 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:42.038483 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:06:42.045406 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:06:42.104761 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:42.105012 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:42.105438 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:42.107401 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:42.116820 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:42.202922 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 14:06:42.233531 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:42.266398 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (372.794s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:42.343473 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:42.398279 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:42.398844 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:42.399195 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:42.413067 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:42.413319 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:42.446383 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:06:42.472195 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 14:06:42.472828 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:42.502689 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:42.503487 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:42.504142 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:42.511461 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:42.517813 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:42.544328 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:06:42.578506 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (373.109s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:42.637797 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:06:42.647984 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:06:42.680478 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (373.208s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:42.700361 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:06:42.744448 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:42.801983 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:42.802629 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:42.803277 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:42.804682 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:42.804749 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:42.899937 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:42.900472 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:42.901315 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:42.914057 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:42.920943 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:42.930789 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:06:42.990936 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:06:43.033762 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (373.565s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:43.097242 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:06:43.179148 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:06:43.188854 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:06:43.198618 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (373.726s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:43.200237 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240503 14:06:43.200559 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:43.201102 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:43.202445 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:43.203864 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:43.203922 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:43.297992 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:43.300602 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:43.303002 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:43.305478 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:43.308714 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:43.309942 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:06:43.319604 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:43.592828 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (374.120s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:43.598367 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:43.599283 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:43.600698 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:43.607496 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:43.608676 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:43.640900 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (374.172s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:43.641893 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240503 14:06:43.695402 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:43.696259 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:43.705132 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:43.705397 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:43.707268 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:43.813042 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:43.962534 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:43.991225 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:43.992239 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:43.996259 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:44.003021 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:44.003769 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:44.073761 30242 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (374.604s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:44.086279 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:44.090622 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:44.093417 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:44.093739 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:44.095430 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:44.102525 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:06:44.109371 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:06:44.119486 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (374.647s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:44.166803 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:06:44.249353 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:06:44.288414 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:44.317409 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:44.385008 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:44.389616 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:44.390630 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:44.397027 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:44.397437 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:44.430001 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:44.493145 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:44.494122 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:44.494819 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:44.500177 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:44.512825 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:44.576423 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:06:44.627169 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (375.154s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:44.651829 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (375.183s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:44.657366 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 14:06:44.657943 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:06:44.738786 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:06:44.751600 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:44.796876 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:44.798911 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:44.799327 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:44.799432 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:44.812279 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:44.895555 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:44.895828 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:44.900871 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:44.908627 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:44.911412 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:44.912143 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:44.973839 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:45.035316 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (375.566s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:45.137630 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:06:45.146898 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:45.155508 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (375.683s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:45.203786 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:45.205281 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:06:45.207432 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:45.208001 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:45.216073 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:45.221227 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:45.259222 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:06:45.305589 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:45.305615 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:45.306565 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:45.307323 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:45.319674 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:45.371958 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:06:45.405740 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 14:06:45.427789 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:45.472905 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:06:45.551311 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (376.082s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:45.606567 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:45.607358 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:45.616861 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:45.617640 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:45.617743 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:45.639969 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:06:45.646852 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:45.675522 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (376.203s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:45.676625 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240503 14:06:45.684247 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:45.710997 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:45.711647 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:45.713801 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:45.721159 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:45.724989 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:45.918359 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:46.010687 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:46.013653 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:46.013989 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:46.023383 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:46.026032 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:46.066200 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (376.597s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:46.067101 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240503 14:06:46.107662 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (376.635s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:46.117533 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:46.120754 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:46.121336 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:46.122740 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:46.137248 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:46.419472 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:46.419844 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:46.421947 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:46.423993 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:46.424837 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:46.441468 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:46.531127 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (377.062s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:46.537600 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:46.539392 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:46.545817 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:46.550024 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:46.563297 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:46.565819 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (377.093s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:46.613754 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:06:46.673381 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:06:46.842083 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:06:46.853072 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:06:46.856555 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:46.858795 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:46.860123 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:46.862022 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:46.874104 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:46.900693 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:06:46.919485 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:46.958755 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:46.959148 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:46.961374 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:46.962975 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:46.963070 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:46.968185 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:46.982704 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (377.514s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:46.998596 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:06:47.096876 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:06:47.124235 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (377.652s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:47.264307 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:47.265864 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:47.266288 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:47.266343 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:47.266764 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:47.336746 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:06:47.352187 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 112 similar messages]
I20240503 14:06:47.356767 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:47.358896 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:47.361368 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:47.362601 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:47.364187 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:47.367406 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:47.385675 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (377.916s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:47.440062 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:06:47.460896 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:47.599043 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (378.126s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:47.621102 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:06:47.622726 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:47.651930 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:47.655871 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:47.656340 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:47.664543 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:47.664800 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:47.715441 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 14:06:47.716177 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:06:47.727327 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:06:47.756522 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:47.757608 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:06:47.759081 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:47.759548 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:47.761021 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:47.766021 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:47.856223 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:47.906069 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (378.437s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:47.972539 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:06:47.975723 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:48.024837 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (378.552s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:48.025728 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240503 14:06:48.054013 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:48.058388 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:48.060731 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:48.067497 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:48.067538 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:48.102766 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:48.143812 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:48.145344 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:48.148103 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:48.156265 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:48.159685 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:48.426900 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (378.954s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:48.432137 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (378.963s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:48.440079 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240503 14:06:48.441102 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:48.446230 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:48.451524 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:48.453729 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:48.454968 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:48.462571 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:48.548339 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:48.549526 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:48.550818 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:48.551827 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:48.563352 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:48.571878 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:48.843474 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:48.844807 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:48.848137 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:48.852403 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:48.852622 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:48.916972 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (379.444s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:48.946100 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:48.947990 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:48.950237 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:48.951422 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:48.951745 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:48.967486 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:49.025849 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:49.043076 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (379.574s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:49.098244 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:49.243435 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:49.245705 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:49.248776 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:49.249207 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:49.258729 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:49.261248 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:49.272327 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 14:06:49.312173 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (379.839s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:49.353879 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:49.356472 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:49.357127 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:49.358847 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:49.360898 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:49.381726 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:49.402480 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:49.467449 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (379.998s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:49.486862 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:49.569952 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:49.667382 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:49.669677 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:49.672426 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:49.677065 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:49.677556 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:49.702592 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:49.745186 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (380.272s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:49.751307 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 14:06:49.752065 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:06:49.761476 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:49.765794 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:49.769110 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:49.771579 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:49.776440 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:49.789979 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:49.794067 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:49.885979 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:06:49.924634 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:49.941040 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (380.472s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:49.958882 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:06:50.050170 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:50.072654 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:50.076051 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:50.080444 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:50.082778 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:50.082849 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:50.106545 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:06:50.162806 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:50.172837 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:50.174135 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:50.175398 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:50.183748 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:50.200160 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:50.303449 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:06:50.315733 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:50.354168 32090 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (380.885s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:50.357023 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (380.884s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:50.357815 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240503 14:06:50.393714 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:50.419548 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:06:50.481185 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:50.484197 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:50.484544 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:50.492326 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:50.495664 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:50.580725 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:50.580749 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:50.581451 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:50.582650 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:50.584362 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:50.664186 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:06:50.800395 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:50.869804 949 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (381.401s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:50.872565 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240503 14:06:50.876467 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:50.882095 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:50.882151 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:50.885850 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:50.898975 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:50.899953 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:50.932961 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (381.460s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:50.988085 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:50.989539 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:50.990417 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:50.995659 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:51.011116 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:51.079762 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:51.283875 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:51.284353 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:51.289458 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:51.292601 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:51.301517 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:51.323700 949 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (381.851s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:51.376506 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:51.381217 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:51.382743 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:51.383425 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:51.394579 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:51.436990 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:51.487108 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (382.018s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:51.533411 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:06:51.647266 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:51.672500 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:51.672565 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:51.674379 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:51.685092 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:51.691907 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:51.727411 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:51.771845 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:51.775930 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:51.776582 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:51.777762 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:51.786597 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:51.795807 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:51.886829 949 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (382.414s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:51.919234 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:51.966174 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:06:51.970105 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:52.031634 29345 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (382.562s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:52.077457 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:52.077795 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:52.084847 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:52.087497 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:52.088186 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:52.171449 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:52.175477 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:52.177003 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:52.177547 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:52.178867 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:52.284267 949 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (382.812s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:52.335948 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:06:52.393582 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 14:06:52.394066 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:52.411420 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:06:52.423869 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:52.469769 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:52.470429 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:52.474059 949 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (383.005s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:52.477535 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:52.478842 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:52.479683 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:52.487418 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:52.488916 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:52.494271 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:52.573017 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:52.578615 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:52.579012 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:52.580336 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:52.587299 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:52.594305 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:52.697239 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:52.757809 949 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (383.285s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:52.758730 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240503 14:06:52.788516 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:06:52.800058 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:52.801327 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:06:52.818531 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:06:52.845463 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 14:06:52.867725 949 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (383.398s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:52.868659 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:52.872066 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:52.872687 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:52.872936 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:52.892894 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:52.905174 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:06:52.969475 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:52.972251 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:52.974138 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:52.980934 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:52.984998 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:53.131525 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:06:53.265837 949 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (383.793s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:53.266973 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:53.268018 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:53.272186 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:53.285454 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:53.286264 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:53.323766 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (383.855s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:53.324594 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240503 14:06:53.367091 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:53.369102 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:53.369690 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:53.370440 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:53.387377 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:53.387576 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:53.646389 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:06:53.660794 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:53.661835 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:53.665246 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:53.675760 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:53.680384 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:53.721705 475 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:06:53.721858 475 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:06:53.722327 475 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:06:53.723130 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:06:53.723289 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:06:53.723356 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:06:53.723491 475 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:06:53.762799 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:53.763338 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:53.763423 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:53.763455 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:53.763797 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:53.827602 1606 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (384.358s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:53.888746 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (384.416s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:53.889537 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:53.949450 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:06:53.967324 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:06:54.057523 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:54.059116 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:54.060151 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:54.062258 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:54.067736 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:54.080489 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:54.152156 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:54.152529 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:54.153643 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:54.155347 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:54.156248 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:54.355940 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:06:54.373813 1606 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (384.905s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:54.444904 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:54.448526 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:54.448617 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:54.455891 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:54.462071 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:54.476253 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:06:54.486752 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:54.504822 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (385.032s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:54.541842 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:54.542140 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:54.545893 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:54.546593 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:54.549705 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:54.625401 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:06:54.629880 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:06:54.671942 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:06:54.793537 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:06:54.808130 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:54.836277 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:54.838481 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:54.840799 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:54.841997 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:54.843346 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:54.928769 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:54.929347 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:54.935253 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:54.935498 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:54.939369 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:54.945008 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:54.951361 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (385.482s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:55.032362 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:55.086429 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (385.614s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:55.117308 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:06:55.162621 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:06:55.176817 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 14:06:55.177511 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:55.228598 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:55.230144 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:55.230567 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:55.230705 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:06:55.233347 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:55.239385 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:55.247699 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:06:55.252318 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:55.326921 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:55.328243 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:55.330940 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:55.335382 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:55.339934 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:55.391374 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:06:55.484885 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:55.499516 1606 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (386.030s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:55.506377 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:06:55.508984 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (386.036s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:55.509971 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240503 14:06:55.530907 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:06:55.621667 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:55.622278 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:55.622555 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:55.625924 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:55.629516 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:55.665297 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240503 14:06:55.716954 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:55.718276 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:55.721022 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:55.723940 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:55.724124 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:55.896757 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (386.428s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:55.897655 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240503 14:06:56.012951 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:56.015755 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:56.016613 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:56.016748 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:56.020108 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:56.027773 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:56.037218 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (386.565s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:56.112308 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:56.113099 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:56.114068 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:56.114930 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:56.120755 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:56.379595 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:06:56.396884 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:06:56.405692 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:56.411101 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:56.412591 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:56.413825 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:56.415220 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:56.424912 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:56.470711 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (387.002s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:56.520815 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:56.520913 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:56.522389 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:56.532018 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:56.538380 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:56.560873 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (387.088s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:56.610885 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:06:56.813123 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:56.813319 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:56.815222 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:56.818832 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:56.821432 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:56.825579 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:06:56.912652 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:56.913698 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:56.916456 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:56.921759 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:56.925478 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:56.933956 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:06:56.991351 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (387.522s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:57.032063 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:57.088826 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:06:57.115056 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (387.642s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:57.194918 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:06:57.211726 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:06:57.215058 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:57.216674 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:57.216665 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:57.224864 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:57.225219 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:57.312139 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:57.314393 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:57.316222 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:57.317732 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:57.329732 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:57.410290 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:06:57.438911 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240503 14:06:57.439543 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:57.478474 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:06:57.491263 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:06:57.500758 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (388.032s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:57.557207 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (388.085s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:57.603776 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:06:57.608860 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:57.609342 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:57.609356 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:57.612560 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:57.625458 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:57.651973 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:06:57.666718 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:06:57.701773 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:06:57.702204 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:57.702443 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:57.707032 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:57.708403 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:57.716032 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:57.723033 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:06:57.795277 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:06:57.808550 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:06:57.887756 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (388.419s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:57.919451 573 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:06:57.919570 573 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:06:57.920115 573 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
W20240503 14:06:57.920550 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 14:06:57.920656 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:06:57.920771 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:06:57.920843 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:06:57.921169 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:06:57.921236 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:06:57.921288 12365 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:06:57.921406 573 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:06:57.936334 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (388.464s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:57.937077 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240503 14:06:58.001593 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:58.002377 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:58.012544 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:58.014994 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:58.016575 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:06:58.027563 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:06:58.063093 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:06:58.071897 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:06:58.102052 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:58.103102 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:58.105115 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:58.107933 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:58.111382 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:58.327803 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (388.859s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:58.328671 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240503 14:06:58.337360 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (388.865s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:58.395989 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:58.396603 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:58.398885 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:58.400300 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:58.405041 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:58.488500 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:58.489132 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:58.489598 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:58.502456 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:58.502719 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:58.666621 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:58.781315 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (389.309s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:58.784570 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:58.786891 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:58.787760 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:58.790362 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:58.791169 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:58.879724 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:58.883342 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:58.885143 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:58.885672 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:58.886129 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (389.417s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:58.886175 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:59.032737 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:06:59.101137 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:06:59.179912 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:59.180603 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:59.180964 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:06:59.185197 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:59.188467 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:59.196869 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:06:59.197554 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:59.272836 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:59.274789 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:59.276345 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:59.276764 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:59.276863 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (389.808s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:59.279136 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:06:59.282006 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:59.314603 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (389.842s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:59.394063 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:06:59.569599 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:59.570716 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:59.572418 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:59.572578 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:59.572981 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:59.594487 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:06:59.656136 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:06:59.662159 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:06:59.667601 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:59.668642 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:59.670315 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:59.690204 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:06:59.697798 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:06:59.702044 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:06:59.702251 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (390.233s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:06:59.712765 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:06:59.800392 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:06:59.891947 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (390.419s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:06:59.958865 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:06:59.959548 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:06:59.961468 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:06:59.962638 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:06:59.969195 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:06:59.973608 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:06:59.974809 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:00.022348 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:07:00.063055 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:00.064229 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:00.071588 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:00.072351 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:00.073503 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:00.100723 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (390.632s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:00.218662 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:07:00.228942 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:07:00.255882 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:07:00.264004 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:07:00.286659 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:00.358597 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:00.362305 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:00.362536 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:00.364817 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:00.367267 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:00.375044 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (390.902s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:00.382714 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:07:00.383339 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240503 14:07:00.472100 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:00.474243 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:00.474516 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:00.478832 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:00.479040 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:00.481703 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (391.013s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:00.482705 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240503 14:07:00.567696 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 14:07:00.572669 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:07:00.584048 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:07:00.721894 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:00.766892 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:07:00.767586 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:07:00.780055 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:00.781606 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:00.782377 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:00.785770 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:00.791774 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:00.874821 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:00.876765 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:00.877396 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:00.884411 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:00.901927 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:00.914920 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (391.442s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:00.961961 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (391.493s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:01.187729 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:01.190340 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:01.192941 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:01.193634 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:01.195422 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:01.300063 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:01.301930 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:01.305099 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:01.309578 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:01.310634 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:01.337870 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:01.409265 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (391.940s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:01.500459 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (392.028s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:01.582868 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:01.605109 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:01.606415 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:01.609406 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:01.616627 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:01.618750 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:01.647524 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:01.705909 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:01.707227 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:01.710570 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:01.712972 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:01.722101 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:01.768172 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:01.793689 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (392.324s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:01.833575 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:01.876034 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:07:01.975943 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:07:02.000494 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:02.012498 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:02.013115 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:02.013599 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:02.022629 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:02.026959 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:02.071957 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (392.599s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:02.119143 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:02.121443 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:02.121699 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:02.122891 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:02.123225 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:02.151984 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:02.200870 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (392.732s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:02.223912 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:07:02.378604 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:07:02.379638 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:07:02.397125 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:02.404911 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:07:02.416914 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:07:02.421878 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:02.425319 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:02.429704 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:02.435649 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:02.436482 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:02.505789 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 111 similar messages]
I20240503 14:07:02.506872 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (393.034s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:02.533242 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:02.536016 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:02.543365 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:02.548998 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:02.555438 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:02.557073 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:07:02.557638 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:07:02.613834 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (393.145s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:02.614858 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 14:07:02.691931 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:07:02.754001 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:07:02.839749 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:02.840438 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:02.843204 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:02.843420 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:02.843892 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:02.849342 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:02.850313 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:07:02.922240 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (393.449s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:02.923421 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240503 14:07:02.942389 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:02.942684 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:02.945439 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:02.946576 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:02.946631 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:03.073338 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:03.117372 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (393.648s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:03.168174 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240503 14:07:03.185536 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:07:03.196879 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:07:03.246389 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:03.252017 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:03.252846 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:03.253090 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:03.262878 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:03.318089 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (393.845s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:03.354463 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:03.354677 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:03.357450 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:03.358069 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:03.359400 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:03.368516 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:03.656908 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:03.657608 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:03.658282 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:03.661165 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:03.661348 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:03.717978 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (394.249s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:03.764508 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:03.768225 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:03.768277 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:03.769732 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:03.770509 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:03.781347 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (394.309s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:03.978899 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:04.068300 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:04.068384 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:04.079704 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:04.080111 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:04.080170 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:04.093014 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:07:04.103646 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:07:04.117302 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:07:04.169639 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:04.174177 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:04.175181 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:04.175830 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:04.184332 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:04.260169 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (394.791s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:04.266111 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:07:04.364357 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:04.370085 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (394.897s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:04.431777 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:04.489615 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:04.490041 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:04.490226 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:04.492692 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:04.494191 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:04.516597 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:07:04.532709 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:07:04.585652 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:04.590137 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:04.592190 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:04.599360 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:04.607592 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:04.659020 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (395.190s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:04.666015 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:07:04.744273 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:04.890439 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:04.893837 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:04.894237 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:04.899852 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:04.915031 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:04.923614 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (395.451s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:04.936694 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:07:04.937393 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:07:04.938052 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:07:04.944828 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:07:04.977217 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:05.000713 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:05.001492 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:05.002951 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:05.004492 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:05.023764 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:05.029537 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:05.047750 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:07:05.061026 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:05.094686 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (395.626s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:05.095814 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 14:07:05.277576 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:07:05.295869 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:05.299804 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (395.827s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:05.300704 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240503 14:07:05.313164 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:05.314960 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:05.315398 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:05.317760 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:05.317850 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:05.415417 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:05.418962 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:05.419229 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:05.419708 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:05.420575 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:05.547530 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:05.550866 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (396.082s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:05.556010 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:07:05.709437 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:07:05.732501 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:05.733170 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:05.733269 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:05.736323 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:05.747967 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:05.802985 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:07:05.808568 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 14:07:05.814072 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:07:05.824956 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (396.352s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:05.837997 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:05.838032 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:05.838553 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:05.839620 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:05.842274 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:05.969941 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:06.127692 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (396.658s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:06.133960 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:06.135967 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:06.140962 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:06.144117 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:06.150624 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:06.239338 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:06.244096 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:06.245985 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:06.246645 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:06.254074 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (396.781s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:06.267714 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:06.433858 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:06.552277 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:06.552973 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:06.553364 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:06.553632 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:06.555969 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:06.564762 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:07:06.588979 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:07:06.612623 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:07:06.638085 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (397.169s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:06.665277 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:06.667759 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:06.673388 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:06.673930 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:06.675880 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:06.823295 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:07:06.832157 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (397.359s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:06.911849 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:06.977290 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:06.979588 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:06.980839 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:06.981578 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:06.988708 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:07:06.997517 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:07.006507 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:07:07.083010 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:07.085178 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:07.085562 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:07.089269 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:07.096446 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:07.138494 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (397.669s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:07.288226 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (397.816s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:07.291042 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:07:07.344209 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:07:07.385419 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:07.387668 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:07.388703 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:07.390213 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:07.392143 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:07.411595 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:07:07.415719 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:07:07.421077 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:07:07.477936 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:07.496711 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:07.498435 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:07.500362 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:07.509999 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:07.512079 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 14:07:07.512758 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:07:07.513144 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:07.561847 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (398.093s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:07.562922 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240503 14:07:07.601861 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:07:07.688402 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:07:07.706879 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:07:07.756065 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (398.283s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:07.757020 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 14:07:07.789445 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:07:07.800956 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:07.801761 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:07:07.804498 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:07.804875 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:07.808576 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:07.815321 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:07.870239 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:07:07.899870 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:07.909775 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:07.910511 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:07.913882 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:07.929266 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:07.962358 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:08.062386 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:07:08.170150 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (398.701s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:08.212719 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:08.213927 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:08.214597 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:08.215328 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:08.217149 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:08.222509 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (398.750s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:08.249696 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:07:08.319689 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:08.323031 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:08.323544 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:08.324463 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:08.324887 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:08.348248 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:08.402129 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 14:07:08.620674 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:08.623747 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:08.626392 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:08.627123 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:08.628254 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:08.731103 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (399.258s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:08.732802 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:08.733922 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:08.735028 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (399.266s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:08.735962 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:08.737108 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:08.751441 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:08.880790 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:08.937319 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:07:09.037846 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:09.039474 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:09.039722 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:09.050415 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:09.056316 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:09.141997 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:09.142004 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:09.143954 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:09.153400 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:09.155074 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:09.156574 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (399.687s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:09.227871 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (399.755s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:09.296556 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:09.301182 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:09.321545 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:09.406479 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:09.445346 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:09.448817 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:09.449000 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:09.451568 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:09.455082 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:09.457779 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:07:09.553372 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:09.556222 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:09.556972 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:09.567962 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:09.575871 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:09.615793 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (400.147s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:09.663007 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (400.190s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:09.700793 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:07:09.704021 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:09.763389 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:07:09.781975 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:09.787308 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:07:09.856899 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:09.860003 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:09.863875 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:09.864995 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:09.885084 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:09.893793 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:07:09.970654 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:09.971190 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:09.973981 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:09.975469 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:09.976313 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:09.988049 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:07:09.989262 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:07:10.017891 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:07:10.018536 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:07:10.028698 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:07:10.104873 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (400.636s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:10.105742 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240503 14:07:10.142699 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:07:10.180509 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (400.708s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:10.181588 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 14:07:10.194643 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:10.262110 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:07:10.277434 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:10.279314 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:10.282300 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:10.290230 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:10.290572 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:10.377260 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:10.378738 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:10.379328 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:10.387071 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:10.402222 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:10.434073 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:10.506065 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:07:10.515373 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:10.634896 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:10.638654 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:07:10.646754 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (401.178s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:10.680512 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (401.208s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:10.691699 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:10.692832 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:10.695345 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:10.695950 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:10.704896 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:10.797017 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:10.797895 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:10.802763 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:10.803372 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:10.811064 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:10.861963 475 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:07:10.862104 475 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:07:10.862555 475 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:07:10.863049 12104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 14:07:10.863765 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:07:10.863842 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:07:10.863901 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:07:10.864037 475 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:07:10.869529 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 14:07:11.083546 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:11.102905 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:11.103780 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:11.103976 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:11.105515 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:11.111634 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:11.155808 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (401.683s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:11.181552 1302 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (401.712s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:11.204833 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:11.206519 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:11.207507 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:11.220744 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:11.223321 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:11.475873 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:07:11.511996 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:11.512647 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:11.517520 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:11.524706 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:11.535753 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:11.618724 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:11.619745 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:11.623631 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:11.631160 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:11.632539 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:11.696411 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:11.752084 5356 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (402.279s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:11.763972 1946 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (402.295s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:11.843605 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:07:11.876611 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:07:11.927274 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:11.929184 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:11.930428 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:11.931528 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:11.932201 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:11.934167 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:07:11.989008 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:07:12.032752 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:12.033157 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:12.033310 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:12.033962 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:12.034508 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:12.065550 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:12.076151 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:12.126420 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:07:12.163331 5356 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (402.694s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:12.181285 5356 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (402.709s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:12.182242 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:07:12.226227 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:12.228102 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:12.289136 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:12.325151 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:07:12.338654 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:12.341157 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:12.341576 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:12.341751 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:12.344574 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:12.365065 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:07:12.444725 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:12.444810 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:12.446449 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:12.453174 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:12.458184 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:12.521246 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:12.551961 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 14:07:12.552686 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:12.680532 5356 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (403.208s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:12.681541 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240503 14:07:12.721886 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (403.253s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:12.722906 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240503 14:07:12.744071 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:07:12.759426 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:12.759727 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:12.761449 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:12.775950 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:12.778273 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:12.817574 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:07:12.828672 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:07:12.865677 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:12.866083 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:12.866808 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:12.868659 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:12.871131 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:12.939601 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:07:12.953227 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:07:13.015674 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:13.159444 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (403.687s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:13.168815 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:13.169616 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:13.169785 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:13.170958 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:13.173420 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:13.231827 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:07:13.268858 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:13.268903 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:13.273568 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:13.275592 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:13.275714 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:13.321674 5356 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (403.852s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:13.539147 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:13.546701 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 14:07:13.573978 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:13.575593 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:13.577924 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:13.579437 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:13.585609 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:13.661551 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (404.189s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:13.676088 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:13.678982 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:13.682463 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:13.682909 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:13.687548 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:13.752460 5356 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (404.283s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:13.982331 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:13.983630 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:13.986181 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:13.987722 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:13.988538 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:14.006192 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:07:14.077003 475 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:07:14.077155 475 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:07:14.077612 475 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:07:14.078611 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:07:14.078989 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:07:14.079056 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:07:14.079249 475 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:07:14.089066 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:14.093904 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:14.094137 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:14.096204 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:14.097262 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:14.103899 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:14.114524 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:07:14.165819 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:07:14.169462 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:07:14.184194 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:07:14.264429 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (404.795s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:14.280970 5356 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (404.808s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:14.347270 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:07:14.390533 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:14.394352 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:14.395330 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:14.398519 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:14.406832 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:14.513513 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:14.515321 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:07:14.515749 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:14.516767 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:14.520258 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:14.523581 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:14.589116 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:07:14.632790 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:07:14.668551 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:07:14.693241 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:14.710554 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (405.241s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:14.710769 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:07:14.742421 5356 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (405.270s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:14.769006 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:07:14.814821 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:14.814960 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:14.815693 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:14.818372 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:14.818939 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:14.921243 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:14.922147 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:14.926545 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:14.928751 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:14.943399 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:14.959141 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:07:14.987501 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:07:15.116832 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (405.648s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:15.117767 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240503 14:07:15.157308 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:07:15.175565 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:07:15.224778 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:15.228778 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:15.229861 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:15.230140 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:15.238205 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:15.242851 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:07:15.284004 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:15.301551 5356 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (405.829s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:15.302502 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240503 14:07:15.329910 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:07:15.333094 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:15.334583 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:15.340809 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:15.349040 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:15.349675 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:15.378023 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:07:15.386193 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:15.550005 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:07:15.550144 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (406.081s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:15.635329 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:15.636632 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:15.638754 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:15.640728 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:15.649513 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:15.744233 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:15.744531 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:15.747421 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:15.747498 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:15.749428 5869 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:07:15.749528 5869 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:07:15.751406 5869 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
W20240503 14:07:15.753343 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:07:15.753518 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:07:15.753566 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:07:15.753791 5869 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:07:15.755385 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:15.755554 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:07:15.755689 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:07:15.755760 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:07:15.789470 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:15.903787 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240503 14:07:15.912330 5356 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (406.440s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:15.984035 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (406.515s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:16.047793 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:16.050874 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:16.053977 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:16.068585 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:16.074239 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:16.156766 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:16.157959 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:16.164036 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:16.164319 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:16.170198 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:16.292999 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240503 14:07:16.293617 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:07:16.319633 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:16.467422 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:16.469527 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:16.469992 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:16.472087 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:16.478566 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:16.503409 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (407.034s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:16.508519 5356 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (407.036s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:16.527437 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:07:16.572856 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:16.576959 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:16.578691 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:16.580076 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:16.580555 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:16.612402 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:16.711666 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:16.726845 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:16.820852 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:16.866842 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:16.870069 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:07:16.877672 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:16.878262 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:16.878547 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:16.882928 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:16.893205 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:16.918083 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (407.449s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:16.984591 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:16.988760 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:16.989708 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:16.992192 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:16.993003 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:16.997097 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (407.524s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:16.997306 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:17.096021 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:07:17.098011 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:07:17.137253 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:07:17.229619 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:07:17.291692 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:17.291692 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:17.293936 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:17.294323 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:17.300798 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:17.312390 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:17.364089 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:07:17.392066 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:17.393983 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:17.399469 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:17.401592 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:07:17.407984 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:17.412899 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:17.422923 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (407.954s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:17.426862 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240503 14:07:17.431577 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (407.959s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:17.512123 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:07:17.594961 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 14:07:17.608083 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:07:17.694520 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:07:17.704437 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:17.709007 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:17.710700 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:17.715387 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:17.724905 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:17.731524 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:07:17.739874 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:17.810341 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:17.811304 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:17.813910 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:17.816093 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:17.824121 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:17.824903 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:17.860934 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:07:17.949191 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (408.476s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:17.949906 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240503 14:07:18.013257 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (408.544s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:18.119668 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:18.119737 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:18.120523 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:18.122730 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:18.129536 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:18.219712 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:18.223960 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:18.225277 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:18.229084 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:18.231389 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:18.355219 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:18.390223 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (408.918s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:18.528688 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:18.529048 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:18.529448 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:18.531044 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:18.535172 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:18.567780 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (409.099s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:18.627080 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:18.627969 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:18.630851 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:18.636245 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:18.639035 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:18.831599 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 23 similar messages]
I20240503 14:07:18.880394 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (409.408s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:18.922582 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:18.930016 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:18.933367 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:18.933378 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:18.936445 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:18.944495 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:18.965409 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (409.496s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:19.009562 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:07:19.032536 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:19.035161 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:19.038832 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:19.043380 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:19.054533 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:19.127386 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:07:19.244395 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:07:19.270177 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:19.281004 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:07:19.306617 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:07:19.313185 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:19.327783 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:07:19.340415 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:19.342715 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:19.342806 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:19.351399 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:19.351687 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:07:19.352814 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:07:19.366823 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:19.373857 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (409.901s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:19.428167 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:07:19.463240 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:19.465037 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:19.469123 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:19.471307 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:19.482360 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:19.555765 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (410.087s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:19.730191 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:07:19.767079 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:19.770484 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:19.770709 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:19.780304 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:19.781826 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:19.800288 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:19.821420 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:19.822784 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:07:19.831436 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (410.359s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:19.872606 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:19.875393 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:19.876650 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:19.885435 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:19.885689 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:19.892092 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:19.953008 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:20.053113 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:20.109166 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:07:20.123661 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (410.654s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:20.124187 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:07:20.124794 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240503 14:07:20.176194 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:20.181329 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:20.183225 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:20.184026 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:20.200379 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:20.286378 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:20.288071 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:20.288803 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:20.295411 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:20.295656 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:20.325690 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:07:20.382172 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (410.909s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:20.383075 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240503 14:07:20.428582 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:07:20.463964 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:07:20.486932 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:20.538028 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (411.069s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:20.589886 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:20.590549 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:20.591540 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:20.597108 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:20.599766 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:20.689386 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:20.694443 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:20.695585 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:20.697329 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:20.720494 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:20.823895 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (411.351s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:20.917537 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 14:07:21.004647 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:21.006911 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:21.010219 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:21.020287 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:21.020668 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:21.049981 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:21.109364 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:21.109607 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:21.110337 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:21.116704 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:21.128149 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:21.161875 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (411.693s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:21.204706 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (411.732s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:21.413436 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:21.413612 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:21.414157 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:21.424068 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:21.426132 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:21.453655 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 14:07:21.454494 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:07:21.520007 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:21.523801 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:21.524606 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:21.537338 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:21.542851 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:21.544126 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (412.075s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:21.599366 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:21.600874 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:07:21.619771 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:07:21.656080 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (412.183s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:21.678604 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:07:21.684188 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:07:21.829461 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:21.829931 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:21.830296 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:21.831712 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:07:21.831939 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:21.833112 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:21.835312 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:07:21.868098 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:07:21.924923 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:07:21.936686 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:21.937332 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:21.939348 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:21.940318 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:21.941885 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:21.994462 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (412.525s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:21.997668 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:22.046370 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:07:22.054891 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:07:22.193240 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (412.721s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:22.236075 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:07:22.236641 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:07:22.237931 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:22.239179 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:22.239326 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:22.239574 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:22.244845 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:22.261718 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:07:22.346925 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:22.348870 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:22.349366 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:22.355815 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:07:22.358022 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:22.361122 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:22.365243 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:07:22.402825 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (412.934s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:22.403915 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240503 14:07:22.557605 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:22.636296 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 111 similar messages]
I20240503 14:07:22.657516 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:22.659878 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:07:22.659931 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:22.663396 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:22.667277 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:22.669040 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:22.771133 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:22.773463 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:22.774966 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:22.782269 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:22.785699 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (413.313s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:22.786764 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240503 14:07:22.787328 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:22.798434 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:07:22.834067 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:07:22.987329 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:07:22.987921 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (413.519s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:23.083225 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:23.083799 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:23.085706 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:23.095556 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:23.100438 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:23.145581 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:23.193703 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:23.193712 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:23.194175 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:23.195147 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:23.195426 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:23.296413 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (413.824s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:23.500188 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:23.501543 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:23.502628 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:23.503283 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (414.034s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:23.504638 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:23.512941 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:23.602612 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:23.603631 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:23.605590 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:23.606897 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:23.607847 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:23.654201 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:23.708961 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (414.236s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:23.909693 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:23.912225 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:23.913331 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:23.914945 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 14:07:23.920485 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:23.928244 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:23.951558 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (414.482s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:23.989607 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:07:24.011725 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:07:24.020578 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:24.023530 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:24.024093 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:24.025700 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:24.031396 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:24.046905 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:07:24.124523 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:24.133785 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:07:24.207274 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (414.734s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:24.230756 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:07:24.240810 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:07:24.332975 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:07:24.337083 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (414.868s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:24.338932 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:24.339383 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:24.340754 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:24.341763 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:24.344033 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:24.360726 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:07:24.408181 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:07:24.440572 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:24.442948 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:07:24.454448 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:24.454465 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:24.462452 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:24.462932 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:24.487846 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:07:24.517196 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:24.531637 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:07:24.618849 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (415.146s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:24.738298 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:07:24.771529 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:24.773310 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:24.774796 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:24.775931 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:24.777006 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:24.843554 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:07:24.873513 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:24.875468 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:24.876332 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:24.877813 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:24.878890 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (415.410s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:24.880250 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240503 14:07:24.883764 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:24.989630 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:25.075914 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (415.603s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:25.076964 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240503 14:07:25.102787 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:07:25.118463 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:07:25.120086 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:07:25.187928 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:25.188278 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:25.190790 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:25.202031 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:25.202183 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:25.237479 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:07:25.256688 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:07:25.290510 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:25.294929 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:25.296433 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:25.297276 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:25.299382 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:25.449200 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:07:25.487990 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:07:25.490129 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (416.021s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:25.541520 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:25.594256 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:25.597740 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:25.598506 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:25.603120 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:25.617445 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (416.145s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:25.618896 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:25.706662 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:25.709632 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:25.712105 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:25.713472 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:25.726542 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:25.940258 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 14:07:26.009397 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:26.014917 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:26.015094 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:26.019665 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:26.025502 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:26.049741 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:26.094611 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (416.622s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:26.104533 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (416.635s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:26.120429 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:26.122357 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:26.122828 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:26.131796 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:26.133176 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:26.225296 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:07:26.365520 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:07:26.428849 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:26.430003 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:26.433382 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:26.436215 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:26.436453 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:26.472947 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:26.536607 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:26.538885 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:26.541715 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:26.542451 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:26.549799 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:26.600991 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:07:26.634491 6553 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (417.162s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:26.636293 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:07:26.640126 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 14:07:26.640708 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:07:26.724233 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (417.255s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:26.803550 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:07:26.819211 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:07:26.838253 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:07:26.850986 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:26.852227 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:26.852245 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:26.852833 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:26.854271 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:26.884212 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:07:26.906735 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:07:26.946333 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:26.949996 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:26.953131 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:26.954581 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:26.955397 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:26.971884 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:07:26.974046 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:07:27.081131 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:27.099092 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:07:27.174764 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:07:27.238126 5527 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (417.765s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:27.260973 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:27.263523 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:27.263743 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:27.272125 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:27.274406 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:27.345708 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (417.877s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:27.361336 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:27.366034 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:27.366986 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:27.368069 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:27.380772 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:27.447532 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:07:27.637168 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:07:27.637929 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:27.646106 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (418.173s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:27.647891 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240503 14:07:27.674492 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:27.675194 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:27.675413 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:27.678145 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:27.686247 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:27.700938 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:07:27.721174 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:07:27.740473 6254 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:07:27.740636 6254 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:07:27.741106 6254 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:07:27.742507 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:07:27.742669 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:07:27.742727 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:07:27.742862 6254 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:07:27.765026 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:07:27.779027 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:27.782624 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:27.784780 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:27.791939 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:27.793587 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:27.826462 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (418.357s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:27.827453 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240503 14:07:27.875650 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:07:28.084715 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:28.084949 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:28.085347 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:28.086279 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:28.088502 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:28.113304 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:28.127696 8566 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (418.655s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:28.186761 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:28.189188 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:28.189213 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:28.189951 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:28.191706 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:28.331169 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (418.862s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:28.377686 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:07:28.497957 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:28.497932 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:28.500262 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:28.500523 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:28.521234 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:28.600932 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:28.601176 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:28.602063 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:28.602595 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:28.608347 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:28.664448 8566 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (419.192s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:28.673238 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:28.689419 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:07:28.720773 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:28.792109 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (419.323s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:28.894968 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:28.894968 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:28.899176 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:28.909433 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:28.910207 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:28.997404 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:28.999961 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:29.000435 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:29.003388 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:29.009341 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:29.038404 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:07:29.049121 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:07:29.132490 11884 maintenance_manager.cc:419] P e301624eacf4431890d4c35d6eed9916: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033390
I20240503 14:07:29.141800 11883 maintenance_manager.cc:643] P e301624eacf4431890d4c35d6eed9916: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.001s sys 0.002s Metrics: {"bytes_written":2007,"delete_count":0,"lbm_write_time_us":150,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":70,"threads_started":1,"update_count":29,"wal-append.queue_time_us":162}
I20240503 14:07:29.142235 11884 maintenance_manager.cc:419] P e301624eacf4431890d4c35d6eed9916: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.132563
I20240503 14:07:29.148927 11883 maintenance_manager.cc:643] P e301624eacf4431890d4c35d6eed9916: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.004s sys 0.003s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15282,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":268,"lbm_read_time_us":180,"lbm_reads_lt_1ms":17,"lbm_write_time_us":204,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":107,"threads_started":2,"update_count":29}
W20240503 14:07:29.167912 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:29.169687 8566 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (419.701s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:29.174844 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:07:29.185189 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:29.231137 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (419.758s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:29.244911 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:29.295504 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:29.299119 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 14:07:29.299914 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:29.301545 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:29.302335 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:29.302453 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:29.304869 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:29.305254 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:07:29.341902 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:07:29.395694 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:29.396039 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:29.396133 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:29.399958 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:29.406658 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:29.445685 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:07:29.486575 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:07:29.531206 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:07:29.579982 8566 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (420.111s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:29.597085 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:29.652341 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (420.180s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:29.652715 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:07:29.703783 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:29.704239 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:29.708370 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:29.708451 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:29.732254 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:29.805691 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:29.809413 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:29.809960 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:29.810966 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:29.811616 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:29.962273 8566 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (420.493s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:29.963392 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240503 14:07:30.025137 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:07:30.071167 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:30.110110 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:30.111042 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:30.114058 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:30.120543 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:30.121745 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:30.167024 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (420.694s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:30.167985 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240503 14:07:30.190784 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:07:30.205107 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:30.205322 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:30.208606 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:30.209478 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:30.217486 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:30.218475 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:30.249114 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:07:30.341454 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:30.349377 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (420.880s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:30.514627 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:30.516883 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:30.517647 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:30.517884 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:30.518086 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:30.601282 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:30.611701 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:30.612918 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:30.613175 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:30.614509 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:30.617296 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:30.636974 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:07:30.652685 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:07:30.753429 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (421.281s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:30.848295 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (421.379s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:30.908066 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:30.908679 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:30.911845 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:30.919430 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:30.921939 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:30.941685 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 14:07:30.999581 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:31.001722 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:31.003609 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:31.009446 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:31.010470 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:31.040535 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:31.150583 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:31.252799 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (421.780s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:31.298748 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:31.301167 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:31.303092 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:31.310217 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:31.310359 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:31.373991 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (421.905s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:31.388290 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:31.390086 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:31.394042 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:31.407650 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:31.407784 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:31.437258 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:07:31.504860 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:07:31.596076 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:31.620468 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:07:31.650074 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:31.707434 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:31.707695 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:31.708026 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:31.710834 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:31.728827 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:31.731616 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:07:31.748728 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:07:31.772423 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:07:31.789983 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 14:07:31.790597 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:31.803731 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:31.806174 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:31.808211 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:31.816383 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:31.816920 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (422.344s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:31.825443 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:31.852401 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:31.903218 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (422.434s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:32.028064 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:32.105422 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:32.108994 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:32.109302 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:32.110845 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:32.116458 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:32.131270 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:32.198292 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:32.201769 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:32.202339 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:32.210116 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:32.211392 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:32.215129 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:32.215621 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:32.308916 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:32.377732 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (422.905s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:32.405239 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:32.466343 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (422.997s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:32.468196 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 14:07:32.498364 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:07:32.504673 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:32.505491 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:32.505741 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:32.509553 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:32.511013 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:32.512383 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:32.611055 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:07:32.614482 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:32.616124 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:32.616427 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:32.616873 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:32.617153 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:32.621941 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:32.683063 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 113 similar messages]
I20240503 14:07:32.792286 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (423.320s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:32.793102 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240503 14:07:32.919770 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:32.921629 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:32.924799 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:32.924994 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:32.934096 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:32.936864 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:32.943658 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:33.003051 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (423.534s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:33.014310 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:33.015512 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:33.015529 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:33.023481 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:33.028379 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:33.217479 9384 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:07:33.217618 9384 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:07:33.218128 9384 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:07:33.218705 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:07:33.218843 12037 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:07:33.218919 12037 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:07:33.219180 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:07:33.219334 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:07:33.219384 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:07:33.219496 9384 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:07:33.310036 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:33.311975 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:33.316622 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:33.323385 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:33.323940 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:33.346143 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:07:33.357267 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (423.885s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:33.393304 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:33.418670 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:33.423718 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:33.426231 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:33.426340 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:33.427403 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:33.474109 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:33.545938 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (424.077s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:33.719447 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:33.722174 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:33.722647 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:33.723101 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:33.725829 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:33.781911 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (424.309s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:33.798044 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:07:33.808547 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:33.813227 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:33.813416 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:33.813841 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:33.816152 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:33.888959 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:33.962342 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:07:34.027287 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:34.086565 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (424.617s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:34.106611 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:34.110157 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:34.110708 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:34.111793 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:34.114284 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:34.181052 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:07:34.204035 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:34.208365 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:34.208467 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:34.209022 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:34.215994 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:34.237021 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:07:34.244415 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:07:34.248980 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (424.776s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:34.294402 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:07:34.357136 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:07:34.438714 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 14:07:34.439359 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:07:34.495651 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:34.506248 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:34.508150 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:34.512310 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:34.512326 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:34.512866 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:34.514467 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:34.524746 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (425.056s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:34.604593 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:34.604938 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:34.606062 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:34.608755 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:34.611699 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:34.613591 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:34.620465 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:34.718627 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:34.742036 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (425.269s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:34.875171 11882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 14:07:34.875872 11882 catalog_manager.cc:4573] TS e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet faa0a6ca00d442eaaf9fca7fc1259ec7: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:07:34.907570 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:34.909582 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:34.910312 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:34.916818 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:34.924031 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:34.967494 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:07:34.992672 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:35.002579 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:35.003319 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:35.007107 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:35.007933 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:35.017560 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:35.020944 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:35.088104 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (425.619s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:35.089150 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240503 14:07:35.217948 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:07:35.236351 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:07:35.262856 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (425.790s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:35.263711 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240503 14:07:35.302366 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:35.302672 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:35.302795 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:35.303653 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:35.306238 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:35.400072 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:35.400887 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:35.403779 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:35.405825 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:35.410976 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:35.467177 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:07:35.592702 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:35.612146 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (426.143s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:35.696156 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:35.697655 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:35.699124 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:35.699316 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:35.701102 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:07:35.702278 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:07:35.706079 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:35.796175 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:35.797278 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:35.797564 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:35.799254 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:35.812037 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:35.831436 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (426.359s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:35.885001 11881 catalog_manager.cc:4573] TS 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet df13ec098dc548afac8c67195d25e7e8: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:07:35.891785 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:07:35.969630 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240503 14:07:36.016479 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (426.547s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:36.092478 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:36.092818 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:36.097069 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:36.106267 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:36.108678 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:36.109951 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:36.144807 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:07:36.194908 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:36.196583 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:36.197191 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:36.199270 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:36.199648 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:36.232985 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (426.760s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:36.392405 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:07:36.465962 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:07:36.501047 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:36.502287 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:36.503772 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:36.506489 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:36.512862 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:36.519665 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:36.575445 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:07:36.583141 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:07:36.588340 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:07:36.591497 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (427.122s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:36.599469 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:36.602708 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:36.603677 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:36.609810 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:36.621819 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:36.720317 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:07:36.732100 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (427.259s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:36.802711 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:07:36.895596 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:36.903291 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:36.908138 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:36.912004 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:36.919117 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:36.997501 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:36.998452 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:36.998838 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:37.000015 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:37.010402 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:07:37.012306 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:37.062152 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:07:37.084579 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 14:07:37.087365 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (427.618s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:37.094475 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:37.146618 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:07:37.291324 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:37.292146 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:37.293411 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:37.297142 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:37.297917 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:37.316727 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:07:37.351636 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (427.879s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:37.389436 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:37.393102 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:37.394218 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:37.397282 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:37.405081 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:37.414397 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:07:37.471055 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:07:37.531538 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (428.062s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:37.532435 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240503 14:07:37.543706 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:07:37.574338 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:07:37.599715 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:37.659068 6254 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:07:37.659201 6254 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:07:37.659588 6254 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:07:37.660568 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:07:37.660694 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:07:37.660739 12103 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:07:37.660849 6254 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:07:37.704418 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:37.705221 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:37.706497 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:37.706580 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:37.708988 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 110 similar messages]
I20240503 14:07:37.711879 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:37.748222 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (428.276s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:37.749141 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240503 14:07:37.797176 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:37.799613 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:37.799631 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:37.803514 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:37.806578 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:37.928870 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:38.008318 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (428.539s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:38.021368 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:38.041625 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:07:38.093115 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:38.099558 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:38.099838 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:38.101640 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:38.108165 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:38.186702 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:38.190326 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:38.190979 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:38.192838 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:38.205566 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:38.247026 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (428.774s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:38.313738 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:07:38.387022 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:07:38.466006 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:38.495363 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:38.496744 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:38.498948 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:38.506713 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:38.507491 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:38.549772 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (429.081s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:38.587910 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:38.588798 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:38.591660 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:38.592263 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:38.592285 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:38.681820 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:38.864840 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (429.392s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:38.870378 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:38.885092 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:38.888324 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:38.889122 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:38.899550 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:38.899538 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:38.963740 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:38.993928 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:38.994575 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:38.997308 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:07:39.000147 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:39.005555 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:39.018282 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:39.073892 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:07:39.077692 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (429.609s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:39.079566 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:39.160916 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:39.270803 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (429.798s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:39.303992 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:39.304216 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:39.306226 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:39.306850 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:39.308374 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:39.339121 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:39.394603 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:39.398182 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:39.398550 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:39.400138 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:39.401110 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:39.402452 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:07:39.477262 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:07:39.519666 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:07:39.526904 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (430.058s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:39.547466 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:07:39.588804 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 14:07:39.620373 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:39.656580 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (430.184s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:39.695145 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:39.697562 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:39.698403 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:39.700268 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:39.702667 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:39.768406 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:07:39.793639 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:39.794901 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:39.795634 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:39.798528 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:39.798537 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:40.034763 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:07:40.047853 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (430.579s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:40.048821 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240503 14:07:40.091411 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:40.093904 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:40.094988 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:40.097679 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:40.101691 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:40.173880 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:07:40.174345 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:40.181947 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:40.182214 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:40.182790 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:40.183951 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:40.190662 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:40.197626 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:40.251464 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (430.779s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:40.252247 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240503 14:07:40.280397 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:40.482120 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:40.484066 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:40.491674 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:40.496003 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:40.498275 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:40.543350 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:07:40.545555 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:07:40.582501 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:40.585826 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:40.586081 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:40.591418 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:40.594542 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (431.125s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:40.597550 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:40.643074 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (431.170s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:40.683583 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:40.697836 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:07:40.739114 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:07:40.740037 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:40.880373 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:40.884258 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:40.885051 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:40.892046 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:40.898267 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:40.973241 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 14:07:40.978428 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:40.980440 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:40.981565 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:40.987879 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (431.519s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:40.989349 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:40.999420 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:41.061705 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (431.589s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:41.117098 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:07:41.160096 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:41.206331 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:07:41.244908 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:41.285739 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:41.287065 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:41.287170 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:41.287667 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:41.295364 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:07:41.300894 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:41.392554 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:41.393108 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:41.393267 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:41.396015 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:41.397303 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:41.482395 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (432.013s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:41.588083 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:41.590785 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (432.118s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:41.654911 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:07:41.689864 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:41.691828 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:41.692564 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:41.696763 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:41.709100 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:41.770524 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:41.786899 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:41.790104 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:41.790215 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:41.790917 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:41.804296 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:41.807804 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:07:41.854946 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:07:41.937969 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:07:41.977123 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:41.979148 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (432.510s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:42.014741 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:42.045521 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (432.573s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:42.087030 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:42.089205 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:42.090006 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:42.093678 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:42.107790 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:42.192914 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:42.193257 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:42.194695 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:42.196199 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:42.199642 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:42.228847 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 14:07:42.265180 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:42.452090 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (432.979s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:42.453014 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240503 14:07:42.474642 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:07:42.475499 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:42.492456 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:42.494359 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:42.496026 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:42.499804 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:42.501441 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:42.515117 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:42.516995 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:07:42.527586 8554 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (433.058s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:42.528530 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240503 14:07:42.528900 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:07:42.586445 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:42.586757 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:42.590657 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:42.590672 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:42.592420 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:42.710443 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:07:42.787842 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 14:07:42.835659 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:42.898169 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:42.903424 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:42.903950 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:42.907682 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:42.926939 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:42.973094 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:07:42.973858 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:43.011197 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:43.012173 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:43.015408 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:43.015707 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:43.017742 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:43.046685 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (433.574s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:43.076007 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:07:43.100556 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (433.631s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:43.144465 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:07:43.314889 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:43.316545 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:43.316946 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:43.317075 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:43.319897 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:43.412700 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:43.413584 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:43.417955 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:43.420387 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:43.423377 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:43.437379 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:43.506980 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (434.034s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:43.598168 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (434.129s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:43.712565 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:43.716778 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:43.718080 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:43.718477 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:43.729585 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:43.745018 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:07:43.751389 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:07:43.808984 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:43.809937 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:43.810343 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:43.810338 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:43.811950 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:43.877103 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:43.902416 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:07:43.975769 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:07:44.003335 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:44.028914 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (434.556s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:44.088495 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (434.619s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:44.102102 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:44.105924 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:44.106666 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:44.107085 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:44.110270 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:44.119627 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:44.121593 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:44.201248 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:44.201339 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:44.201742 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:44.212103 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:44.214239 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:44.324041 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:07:44.341905 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:44.500319 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:44.505535 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:44.509133 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:44.510437 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:07:44.516094 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:44.516492 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:44.536099 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (435.063s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:44.536545 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:44.599583 11648 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:07:44.599714 11648 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:07:44.600136 11648 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:07:44.601851 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:07:44.602576 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:07:44.602646 12103 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:07:44.602855 11648 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:07:44.607034 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:44.609638 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (435.140s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:44.610670 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:44.610759 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:44.616029 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:44.627380 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:44.679425 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:07:44.816843 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 14:07:44.875164 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:07:44.910398 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:44.913686 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:44.914326 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:44.916018 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:44.916508 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:44.925251 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:44.932165 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:45.013991 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:45.015965 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:45.017707 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:45.019891 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:45.024783 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:45.048884 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (435.576s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:45.049880 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240503 14:07:45.085577 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:45.130246 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (435.661s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:45.131001 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240503 14:07:45.234992 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:07:45.241549 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:45.313803 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:45.314956 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:45.315644 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:45.320245 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:45.322288 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:45.364943 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:07:45.404933 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:45.405095 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:45.411621 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:45.412227 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:45.413034 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:45.428582 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:07:45.514694 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:45.536614 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (436.064s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:45.556113 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:07:45.692426 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (436.223s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:45.711593 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:45.712806 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:45.712930 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:45.714183 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:45.714779 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:45.807951 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:45.808651 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:45.808809 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:45.810237 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 14:07:45.821007 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:45.821059 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:45.832484 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:07:45.952363 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:46.052755 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 14:07:46.106290 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (436.637s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:46.107003 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:46.109090 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:46.111239 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:46.112156 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:46.136699 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:46.152746 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (436.680s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:46.170214 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:46.213274 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:46.216725 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:46.220206 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:46.222148 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:46.227957 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:46.291051 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:07:46.422778 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:46.527798 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:46.527890 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:46.529954 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:46.530695 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:46.531374 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:46.534823 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:46.535059 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (437.062s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:46.544404 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:07:46.562469 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:07:46.594748 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:07:46.600862 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:46.612313 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (437.143s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:46.628850 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:46.632360 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:46.633534 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:46.637064 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:46.646466 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:46.774972 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:46.918668 11648 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:07:46.918772 11648 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:07:46.919155 11648 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:07:46.921030 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
W20240503 14:07:46.921105 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
I20240503 14:07:46.921157 12103 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:07:46.922116 11648 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:07:46.933923 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:46.933990 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:46.940194 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:46.946604 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:46.947548 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:46.957232 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:46.991070 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (437.522s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:47.006778 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:47.029834 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:47.030757 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:47.032366 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:47.041082 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:47.048100 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:47.048660 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:47.070214 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:47.152817 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (437.680s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:47.338534 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:47.340272 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:47.341990 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:47.343221 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:47.351526 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:47.407058 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:47.438282 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:47.440654 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:47.441534 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:47.443482 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:47.443953 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:47.447036 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (437.978s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:47.448182 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240503 14:07:47.495311 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:07:47.606222 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:47.625550 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (438.153s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:47.626307 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240503 14:07:47.728879 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:47.750564 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:47.751574 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:47.752327 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:47.752693 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 14 similar messages]
I20240503 14:07:47.754710 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:47.768110 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:47.794935 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:07:47.802445 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:47.852304 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:47.855226 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:47.858017 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:47.862831 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:47.864126 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:47.961532 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 14:07:48.002983 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:48.050022 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (438.581s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:48.057736 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:07:48.058199 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:07:48.086220 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (438.613s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:48.145877 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:48.146152 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:48.149227 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:48.156872 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:48.157025 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:48.166771 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:48.232748 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:48.233310 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:48.234802 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:48.235188 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:48.248697 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:48.450121 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (438.981s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:48.490437 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:07:48.513358 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:48.530817 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:48.531317 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:48.535104 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:48.545173 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:48.549976 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:48.628129 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:48.628626 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:48.631513 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:48.632627 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:48.634862 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:48.635713 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (439.163s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:48.636580 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:48.719971 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:48.878613 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (439.409s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:48.932863 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:48.937619 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:48.938382 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:48.938995 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:48.947852 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:48.982054 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:49.030512 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:49.031157 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:49.031728 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:49.033166 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:49.035395 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:49.074165 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:49.079428 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:07:49.079761 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:07:49.114809 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:49.172271 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (439.700s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:49.330619 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:49.331496 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:49.332355 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:49.332901 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:49.334290 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:49.335449 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:49.377023 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:49.429013 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:49.429101 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:49.432497 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:49.433072 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:49.445489 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:49.467350 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:49.481874 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (440.013s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:49.554757 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:07:49.581979 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:07:49.690714 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:49.736197 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:49.741356 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:49.748286 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:49.751214 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:49.755668 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:49.772028 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (440.299s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:49.819188 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:07:49.828016 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:49.840776 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:49.841305 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:49.843950 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:49.852936 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:49.855126 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:50.009756 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (440.541s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:50.010725 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240503 14:07:50.031728 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 14:07:50.071322 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:07:50.072131 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:50.137092 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:50.137190 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:50.141841 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:50.142625 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:50.143445 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:50.147974 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (440.675s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:50.155519 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240503 14:07:50.232317 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:50.236799 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:50.237390 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:50.237476 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:50.256047 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:50.392408 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (440.923s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:50.447468 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:50.458989 13833 raft_consensus.cc:482] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e85b0283a0fd4e3ba81b4170f3cdacb5)
I20240503 14:07:50.459110 13833 raft_consensus.cc:504] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c63c6f551949f6a1d7c07760824361" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 33927 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "623e8dcb58544e558207d8e652dfed5f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36489 } }
I20240503 14:07:50.459709 13833 leader_election.cc:288] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845), 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), 623e8dcb58544e558207d8e652dfed5f (127.0.99.129:36489)
I20240503 14:07:50.460214 12035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c06806a19e4f4592d26ed4acc651cd" candidate_uuid: "84c63c6f551949f6a1d7c07760824361" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "623e8dcb58544e558207d8e652dfed5f" is_pre_election: true
I20240503 14:07:50.460347 12035 raft_consensus.cc:1789] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.6
I20240503 14:07:50.460425 12035 raft_consensus.cc:2454] T f5c06806a19e4f4592d26ed4acc651cd P 623e8dcb58544e558207d8e652dfed5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c63c6f551949f6a1d7c07760824361 in term 2.
W20240503 14:07:50.460826 12365 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:07:50.460963 12366 leader_election.cc:334] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:07:50.461033 12366 leader_election.cc:302] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 623e8dcb58544e558207d8e652dfed5f, 84c63c6f551949f6a1d7c07760824361; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:07:50.461158 13833 raft_consensus.cc:2726] T f5c06806a19e4f4592d26ed4acc651cd P 84c63c6f551949f6a1d7c07760824361 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:07:50.535588 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:50.535856 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:50.540269 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:50.541525 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:50.558252 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:50.569342 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:07:50.591428 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:50.623376 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (441.151s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:50.640837 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:50.641427 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:50.643533 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:50.648996 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:50.654196 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:50.659453 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:50.692868 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:50.857678 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (441.388s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:50.892786 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:07:50.938784 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:50.939800 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:50.942905 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:50.951905 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:50.952795 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:50.976145 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:07:50.976964 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:07:51.012308 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:51.039083 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:51.039886 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:51.048564 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:51.050136 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:51.055358 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:51.062357 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (441.590s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:51.089785 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:51.236215 11648 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:07:51.236328 11648 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:07:51.236693 11648 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:07:51.237630 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:07:51.237709 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111)
I20240503 14:07:51.237769 12104 leader_election.cc:302] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b17d706d2344dbaae13cda9b0733143; no voters: 1b4a61e8a11440e7a9f4b300d2442f75, e85b0283a0fd4e3ba81b4170f3cdacb5
I20240503 14:07:51.237890 11648 raft_consensus.cc:2726] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:07:51.290534 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 14:07:51.339670 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:51.343761 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:51.350615 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:51.351177 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:51.351696 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:51.391806 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:07:51.416064 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:51.430723 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (441.961s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:51.432966 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:51.433590 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:51.435912 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:51.436424 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:51.437167 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:51.489024 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (442.016s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:51.621538 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:07:51.629861 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:51.683753 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:51.731292 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:51.735224 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:51.736169 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:51.737032 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:51.738577 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:51.826282 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:51.827646 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:51.831022 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:51.833802 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:51.849570 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:07:51.849896 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:51.850620 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:51.853003 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:51.976725 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (442.508s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:51.978008 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (442.505s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:52.078927 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:52.124753 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:52.128841 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:52.131116 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:52.135803 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:52.137724 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:52.138083 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:52.180583 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:52.216408 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:52.216488 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:52.218822 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:52.225318 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:07:52.225620 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:52.235404 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:52.408834 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:52.444507 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (442.972s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:52.445499 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240503 14:07:52.482342 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:07:52.518052 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:52.519172 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:52.520330 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:52.522708 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:52.529886 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:52.530200 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:52.539764 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (443.070s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:52.540689 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240503 14:07:52.596165 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:52.613751 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:52.615130 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:52.615557 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:52.616976 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:52.623409 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:52.682622 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:07:52.778779 12103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240503 14:07:52.917953 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:52.921064 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:52.923707 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:52.924585 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:52.932703 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (443.463s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:52.943542 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:52.945017 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (443.472s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:52.953866 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:07:52.993993 12234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:07:52.995631 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:07:53.017815 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:53.020522 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:53.023804 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:53.030339 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:53.036968 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:53.198676 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:53.250908 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:53.321409 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:53.326890 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:53.327013 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:53.327644 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:53.329293 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:53.369294 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:07:53.388855 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:53.419720 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:53.420222 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:53.423949 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (443.955s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:53.430687 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:53.432050 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:53.432300 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:53.500001 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (444.027s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:53.534821 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:07:53.717394 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:53.718250 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:53.721752 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:53.735595 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:53.735653 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:53.737505 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:53.817181 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:53.820602 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:53.823400 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:53.829190 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:53.829308 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:53.867923 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (444.399s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:53.912397 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:07:53.956993 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (444.484s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:54.028641 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:54.117482 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:54.121364 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:54.127390 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:54.128598 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:54.129436 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:54.144322 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:07:54.154342 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:54.161397 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:07:54.168900 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:07:54.187309 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:54.213505 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:54.217381 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:54.217933 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:54.218039 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:54.218394 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:54.364199 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (444.895s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:54.476791 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:07:54.522210 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:54.522378 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:54.533681 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:54.540515 11972 consensus_peers.cc:485] T faa0a6ca00d442eaaf9fca7fc1259ec7 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:54.540681 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:54.540912 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:54.543915 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (445.071s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:54.591285 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:54.620102 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:54.621196 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:54.622653 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:54.624297 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:54.636760 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:54.722159 12234 consensus_peers.cc:485] T adf2c44328e244779955e712b99f8331 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:07:54.758143 12365 consensus_peers.cc:485] T 11a8c270532947ba9b505aace44b56f6 P 84c63c6f551949f6a1d7c07760824361 -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:07:54.764958 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (445.296s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:54.766039 12235 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Couldn't send request to peer e85b0283a0fd4e3ba81b4170f3cdacb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46845: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240503 14:07:54.916890 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:54.918895 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:54.930617 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:54.931973 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:54.932708 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:55.009969 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (445.537s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:55.011547 12234 consensus_peers.cc:485] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240503 14:07:55.019888 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:55.020779 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:55.022672 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:55.024113 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:55.038657 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:55.067284 12234 consensus_peers.cc:485] T 2089eeda1e0f4ff5926584d472e6631d P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:07:55.176029 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
W20240503 14:07:55.201210 11972 consensus_peers.cc:485] T dcdc3ae5232a49cdb003d9a2e3cf6d92 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:07:55.210099 11972 consensus_peers.cc:485] T 7d2e1517a8ac4041bfb4b90634cb0fa6 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:07:55.222875 12365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 14:07:55.233100 12365 consensus_peers.cc:485] T 70ffc5888a5945d6a8e166420e6f1bd9 P 84c63c6f551949f6a1d7c07760824361 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:55.293032 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (445.824s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:55.316709 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:55.318447 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:55.319026 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:55.319654 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:55.328786 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:55.409766 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:55.410954 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:55.411574 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:55.413446 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:55.417452 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:55.419467 13710 consensus_peers.cc:485] T 355b5456887b4c5694badc9d53b1809a P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:55.622210 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (446.149s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:55.630916 12234 consensus_peers.cc:485] T d1e9155230c74e92b6ee441de713a419 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:07:55.661844 13710 consensus_peers.cc:485] T 5e6ca408c4254678b67a74f49a34881b P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:07:55.703521 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:55.707996 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:55.709324 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:55.710397 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:55.714385 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:55.714815 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (446.246s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:55.717550 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:55.798980 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:55.800137 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:55.800904 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:55.801266 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:55.801585 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:55.946689 11972 consensus_peers.cc:485] T 0e442bb664f44887aa8afdb62e1d97fc P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:07:56.033094 12103 consensus_peers.cc:485] T 6c4f3bc64e7145e2b692beab730d1496 P 8b17d706d2344dbaae13cda9b0733143 -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:07:56.053764 13710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:07:56.054406 13710 consensus_peers.cc:485] T 92506cde7b084b14b88d4a915731eeb3 P 66190085d1c04e068d5c0dec9c4bf07d -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:07:56.092317 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:56.097479 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:56.098088 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:07:56.099848 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:56.104879 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:56.108158 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:56.192692 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:56.195605 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:56.198421 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:56.199396 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:56.199944 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:56.220183 11807 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (446.751s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
I20240503 14:07:56.221527 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (446.749s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:56.361143 11972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:07:56.424077 11972 consensus_peers.cc:485] T d837845c6f644bca83315c9f7c45246d P 623e8dcb58544e558207d8e652dfed5f -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:07:56.492058 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:56.495867 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:56.495882 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:56.505512 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:56.505759 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:07:56.525979 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:07:56.557176 11972 consensus_peers.cc:485] T a4b4e16598b740cb9d690c2442a3e994 P 623e8dcb58544e558207d8e652dfed5f -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:07:56.568578 12234 consensus_peers.cc:485] T 79b21b1e5292488aad442060b417f3c1 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:07:56.593542 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:56.594607 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:56.594885 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:56.604827 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:56.607153 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:56.610811 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer e85b0283a0fd4e3ba81b4170f3cdacb5 in term 1 (Leader has been unable to successfully communicate with peer e85b0283a0fd4e3ba81b4170f3cdacb5 for more than 300 seconds (447.142s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:56.629739 12234 consensus_peers.cc:462] T c8933689df2c45b59a9cd16024ddb7d9 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111)
I20240503 14:07:56.720259 9180 raft_consensus.cc:952] T c2e9f6eecbec45988b290775277dbddc P ca5386f84994496fa5376624391728ee [term 1 LEADER]: Processing failure of peer 1b4a61e8a11440e7a9f4b300d2442f75 in term 1 (Leader has been unable to successfully communicate with peer 1b4a61e8a11440e7a9f4b300d2442f75 for more than 300 seconds (447.248s)): There is already a config change operation in progress. Unable to evict follower until it completes. Doing nothing.
W20240503 14:07:56.781200 12234 consensus_peers.cc:485] T a6a89230883347d4a64d96d321247012 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:07:56.795922 12234 consensus_peers.cc:485] T b031f09b97664d1a94c14f0661181899 P ca5386f84994496fa5376624391728ee -> Peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Couldn't send request to peer 1b4a61e8a11440e7a9f4b300d2442f75. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:07:56.844245 12234 consensus_peers.cc:485] T a0106f7a40cb467eb4738890ac4e22d4 P ca5386f84994496fa5376624391728ee -> Peer b67f337e10994234baedbf0b19c13513 (127.0.99.134:37075): Couldn't send request to peer b67f337e10994234baedbf0b19c13513. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:37075: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:07:56.888461 12408 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:56.889277 12148 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:07:56.890887 12015 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:07:56.895373 13755 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:07:56.916098 12277 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:07:56.965220 11648 raft_consensus.cc:482] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b4a61e8a11440e7a9f4b300d2442f75)
I20240503 14:07:56.965338 11648 raft_consensus.cc:504] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b17d706d2344dbaae13cda9b0733143" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36263 } } peers { permanent_uuid: "1b4a61e8a11440e7a9f4b300d2442f75" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 34763 } } peers { permanent_uuid: "e85b0283a0fd4e3ba81b4170f3cdacb5" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46845 } }
I20240503 14:07:56.965755 11648 leader_election.cc:288] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763), e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845)
W20240503 14:07:56.966737 12103 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1b4a61e8a11440e7a9f4b300d2442f75 (127.0.99.136:34763): Network error: Client connection negotiation failed: client connection to 127.0.99.136:34763: connect: Connection refused (error 111)
W20240503 14:07:56.966944 12104 leader_election.cc:334] T 211de6f35a844e278e8c1eb1621e7c5f P 8b17d706d2344dbaae13cda9b0733143 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e85b0283a0fd4e3ba81b4170f3cdacb5 (127.0.99.135:46845): Network error: Client connection neg